Networking basic terms

1. NAT(Network Address Transformation) is used to translate "Private IP', which is valid only in LAN into "Public IP" which is valid in WAN.

Front End optimization (FEO)

1. Embed thumbnail of the video instead of the video itself.  
The embedded video not only increases the byte size of your web pages but the browser has to make multiple HTTP requests to render the video player. This increases the overall loading time of your page thus affecting the page speed score. The other drawback with the default YouTube embed code is that it isn’t responsive. If people view your website on a mobile phone, the video player may not resize properly for the small screen.
Refer -  https://www.labnol.org/internet/light-youtube-embeds/27941/

What is SYN_SENT socket status?

When dealing with Network issues one of the linux command that comes handy is 'netstat’

 netstat -an or netstat -an | grep “remote ip”

This command shows all the sockets in the system. Each socket has various status. For example, a socket can be in ‘ESTABLISHED’ status or in ‘LISTENING’ status.

One important status we may come across is ‘SYN_SENT’. When we see a socket in this status, it most probably indicates a Firewall issue, i.e the remote host you are trying to reach is NOT reachable due to a firewall block.

Note that the SYN_SENT status will not remain for long time. It only lasts for couple of seconds. So, you have to be quick in running the netstat command (perhaps in another terminal window)

When the client initiates a connection to Server, it first sends a SYN package. At this point the socket status changes to ‘SYN_SENT’. If the remote server is reachable and working, the client will receive a ‘SYN + ACK’, for which the client will send a ‘ACK’ and thus forms a TCP connection.

How to save cost in AWS? (WIP)

1. Stop all the instances that are not required.
2. Select appropriate EC2 instance
3. Reserve instance (May not be required for Non-production servers if they can be stopped when not required)
4. Select appropriate storage type to store different type of data
5. Use the  advisor available in the AWS portal
6. Business support can be taken only for production and non-production can be without any support.
 

Types of Monitoring

1. Domain monitoring (To keep an eye on the domain expiry for renewal)
2. Device Monitoring (IPS, Firewall, Switch, F5 Load Balancer, etc)
3. Operating System/ VM monitoring (CPU, Memory, I/O, Disk Utilization, etc)
4. Server monitoring (JVM, GC cycles, Mbeans, etc)
5. Architecture monitoring (Real time Deployment architecture monitoring)
6. Service monitoring (URL monitoring)
7. Application monitoring (Slow Transactions)
8. Log monitoring(End user activity)

Cross Joins

Many SQL books and tutorials recommend that you “avoid cross joins” or “beware of Cartesian products” when writing your SELECT statements, which occur when you don't express joins between your tables.  It’s true that you need to ensure that your join conditions are adequately stated so that you don’t accidentally produce this effect, but it is not true that you should avoid these types of joins in every situation.

Cross Joins produce results that consist of every combination of rows from two or more tables.  That means if table A has 3 rows and table B has 2 rows, a CROSS JOIN will result in 6 rows.  There is no relationship established between the two tables – you literally just produce every possible combination.

The danger here, of course, is that if you have table A with 10,000 rows and Table B with 30,000 rows, and you accidentally create the product of these two tables, you will end up with a 300,000,000 row result -- probably not a good idea.  (Though it is great for creating test data and the like.)

Application Performance Monitoring

Monitoring of software application usually includes the following and more -
1. Infrastructure instance monitoring like CPU, Memory, Disk I/O, Network, file descriptors, etc
2. Container (Server) monitoring like JVM utilization, server/access/error logs, etc
3. Application monitoring like No of transactions, type of transactions, response times, etc

There are various tools available for monitoring Application. In this article am sharing some practical benefits of using an APM tool (Appdynamics) which I have been experiencing -


1. Ease of installation, administration and usage

2. Provides information on time taken by the application server and time taken by the database server

3. Provides an ability to further drill to exact method, line number and query that causes slowness

4. Ability to quickly find issues related to lock wait time outs, java exceptions, database exceptions, etc

5. Ease of doing before and after optimization analysis to understand impact on response times of business transactions

6. Provides call stack trace, exceptions, slow queries, query count/repetitive queries, API call count, slow async calls, external calls at one place in easy to understand manner which helps the production management personnel to connect with developers

7. Very less overhead on the application servers on which the java and machine agents are deployed

8. Ability to correlate heap usage with the transactions that impacts it

9. Ease in configuration of thresholds, alerts, package includes and excludes, etc.

10. Helps to find root cause of transaction slowness or run time exceptions in code which can be either in a developers code or the code referred/developed by other teams. Appdynamics helps the team to focus on the issue in hand than finger pointing.

11. Ease of finding the issue saves time and effort that otherwise is spent doing focused performance testing. (Last but not the least, its important to mention that excessive dependency on Appdynamics is not good. A Developer/performance tester some times forgets his very job and depends solely on a APM like Appdynamics) 

Java Issues : java.lang.OutOfMemoryError: GC overhead limit exceeded

In many Cases , we see Out Of Memory Issues When Working with Java Applications on the Java Servers.

“GC overhead limit exceeded” is generally thrown by the Garbage Collectors that we use for the JVM. This is generally thrown by serial or parallel collectors.

The issue comes when more amount of time is spent in doing the Garabage Collection and very less than 2% of the heap is recovered.

This may be due to applications that are running very long time or may be due to threads that are struck. Due to these sort of threads ,the objects that are loaded are not reclaimed and are hold-ed by these struck threads for a long time

The serial or parallel collectors throw this exception and this feature is designed to prevent applications from running for an extended period of time while making little or no progress because the heap is too small. If there is unintended object retention , we need to check code for changes If the retention looks normal, and it is a load issue, the heap size would need to be increased.

Is it important to add index on columns of a table with less no. of records?

I was asked a question regarding the need to add an index on a database table with less no. of records. Say small master tables. I believe its essential to put an index on appropriate columns for such  small tables as well. Here is the rationale behind it -

1. In my experience one of the top root causes of slow transaction is repetitive queries. It can so happen that smaller tables are queried many times which then increases the record scan that many times.

2. The db optimizer chooses an execution plan basis Selectivity index of columns, indexes, PK, data type and some other constraints of the tables involved in the query. I have come across few instances where large queries are tuned just by adding indexes in some master tables the query joins with.

The redo and undo logs

Redo Logs -

Instead of writing to one file (the permanent table records file) you are writing to two different files (the redo log file and the permanent table records file). The difference is that the writes to the database table file(s) are random in nature while the writes to the redo log file will be sequential, which is usually going to be much faster. You perform the much-faster writes to the redo log as needed and then perform the slower writes to the table files periodically when there is time. Thus, the system actually operates faster writing to both files rather than only one file. This applying of the redo logs occurs automatically on startup of the MySQL server after a crash. Very similar processes are used by other database platforms.

Undo Logs -

In addition to the redo log there must also be undo logs. When a database user starts a transaction and executes some commands, the database does not know if the user will end the transaction with a COMMIT or a ROLLBACK command. Ending with  a COMMIT means all the changes made in the course of the transaction have to be preserved (fulfilling the Durable aspect of ACID). If the transaction gets interrupted for some reason, such as the MySQL daemon crashing, the client disconnecting before sending a COMMIT, or the user issuing a ROLLBACK command, then all changes made by the transaction need to be undone.
If the server crashed, the redo log files are applied first, on start up. This puts the database in a consistent state. Now the database server will need to roll back the transactions which were not committed but had  already made changes to the database. Undo logs are used for this.  As an example, if you are running a transaction that adds a million rows and the server crashes after eight hundred thousand inserts are performed, the server will first use the redo log to get the database server into a consistent stand than then will perform a rollback of the eight hundred thousand inserts using the undo logs. For InnoDB, this undo information is stored in the ibdata file(s).

Refer - 

High VM Memory analysis and lessons learnt

This week I investigated a high VM memory utilization on a JBoss servers in production
Using the Linux top command found that the JBoss - Java process was hogging 6.8 GB (Resident Memory). We have set JVM to 6144MB JVM.
Lets look at the sizing formula - 

Max memory = [-Xmx] + [-XX:MaxPermSize] + number_of_threads * [-Xss]

6144 MB + 384 MB + 25 MB (Say 100 threads * 256 K = 25mb) = 6553 MB (Close to 6.8 GB) +
Read this article -

https://dzone.com/articles/why-does-my-java-process which says - But besides the memory consumed by your application, the JVM itself also needs some elbow room.


This means the actual memory the JBoss uses is not equal to the JVM set, but its more at least in case of JAVA 1.7.

One more thing I learnt in investigation. There were several other java processes related to machine agent. We use a APM tool and this tool has a  machine agent which is used to collect the VM resources for correlation Recently the script to restart the JBoss servers were changed from killall -9 java to kill -9 . Due to this for some reason the machine agent processes were left hanging for each day. Each process was hogging 50-60 MB which over days becomes significant given that OS also requires memory.I wonder what would come out as the root cause of the Application servers swapping. It would sound something like this - " The issue of slowness/servers crashing/swapping was due to a APM tool agent" :)

Quality of Service and Throttling (Work in progress)

QoS is not just about isolation; it’s about giving customers/apps what they need.
After setting a QoS its possible that the user experience is bad for that we throttle the incoming traffic so that the experience is maintained of those users who are say already logged in

What to ELK (work in progress)

User Activity Data
How many reports ?
How many Logins ?
How many transactions done?

Host Activity Data
What is the CPU utilization ?
What is the Memory utilization ?

Application Activity Data

Which reports are being accessed most ?


Server logs - Exceptions logging/Excessive logging

Top slow and high count URLs
Respond code wise name and count of URLs
Response time wise (bucket) name and count of URLs
IP wise count of URLs


Security Infrastructure Data-
How many failed logins ?
How many logins ?

Database-
Slow Query Logs

Real User Monitoring-
Page Render
Page Load
Page Weight
FTTB
Page requests
First Interactive Time
Speed Index.

Definitions of RPO and RTO

RPO (Recovery Point Objective) refers to the amount of data at risk. It's determined by the amount of time between data protection events and reflects the amount of data that potentially could be lost during a disaster recovery. The metric is an indication of the amount of data at risk of being lost.

What does it cost for you to lose 1 hour, 4 hours, 8 hours, 1 day of data? (This is the RPO)

The cost of data loss may be more important than the availability of the application. You may want to invest more in the solution to lower the RPO then to invest in the RTO. Maybe data doesn't change much over at the course of a day, so the solution here may be more simplistic. Again, the answer is not as simple as saying "no data loss."

RTO (Recovery Time Objective) is related to downtime. The metric refers to the amount of time it takes to recover from a data loss event and how long it takes to return to service. RTO refers then to the amount of time the system's data is unavailable or inaccessible preventing normal service.

What does it cost for your application to be unavailable for a 1 hour, 4 hours, 8 hours, 1 day? (This is the RTO)
How can you know what is a good price to pay for a DR solution if you don't know how much it costs you to be down? DR is an insurance policy for your application. You need to know how much to pay to protect the investment (your application). It's not as simple as saying "no downtime."

Linux Memory Usage

Linux free -m

The most common way you'll see on the web to check for free memory in Linux is by using the free command.
Using the free -m command to check your Linux memory usage, displays the values as MB instead of KB.

root@server [~]# free -m
               total    used    free  shared  buffers cached
Mem:            1024    1022     1        0      0     822
-/+ buffers/cache:       200   823
Swap:              0       0      0

Most people will run this command and panic thinking they only have 1 MB of free memory on the server:

The free column beside -/+ buffers/cache with 823 MB is the actual free memory available to Linux.
1024 MB is the total system memory available, which would be physical RAM.
1 MB and 823 MB both show free because an application has access to both for memory storage.
1 MB free plus the 822 MB cached gives the 823 MB of memory actually free to use if needed.

High bit rate video delivery (Non-streaming)

In case of video delivery, especially where it isn't streamed its important to keep an eye on the bit rate which governs the file size and hence the download time before which the user would be able to see the video.

For a video length of say 3 mins following would be the size for different fixed bit rates - 1.25 and 5 Mbps -

1.25 Mbps * 3 * 60 =  225/8 = 28 MB
5 Mbps * 3 * 60 = 900/8 = 112.5 MB

File size = bitrate x duration

What is virtual memory, how is it implemented, and why do operating systems use it?

Real, or physical, memory exists on RAM chips inside the computer. Virtual memory, as its name suggests, doesn’t physically exist on a memory chip. It is an optimization technique and is implemented by the operating system in order to give an application program the impression that it has more memory than actually exists. Virtual memory is implemented by various operating systems such as Windows, Mac OS X, and Linux.
So how does virtual memory work? Let’s say that an operating system needs 120 MB of memory in order to hold all the running programs, but there’s currently only 50 MB of available physical memory stored on the RAM chips. The operating system will then set up 120 MB of virtual memory, and will use a program called the virtual memory manager (VMM) to manage that 120 MB. The VMM will create a file on the hard disk that is 70 MB (120 – 50) in size to account for the extra memory that’s needed. The O.S. will now proceed to address memory as if there were actually 120 MB of real memory stored on the RAM, even though there’s really only 50 MB. So, to the O.S., it now appears as if the full 120 MB actually exists. It is the responsibility of the VMM to deal with the fact that there is only 50 MB of real memory.

The paging file and the RAM

Now, how does the VMM function? As mentioned before, the VMM creates a file on the hard disk that holds the extra memory that is needed by the O.S., which in our case is 70 MB in size. This file is called a paging file (also known as a swap file), and plays an important role in virtual memory. The paging file combined with the RAM accounts for all of the memory. Whenever the O.S. needs a ‘block’ of memory that’s not in the real (RAM) memory, the VMM takes a block from the real memory that hasn’t been used recently, writes it to the paging file, and then reads the block of memory that the O.S. needs from the paging file. The VMM then takes the block of memory from the paging file, and moves it into the real memory – in place of the old block. This process is called swapping (also known as paging), and the blocks of memory that are swapped are called pages. The group of pages that currently exist in RAM, and that are dedicated to a specific process, is known as the working set for that process.
As mentioned earlier, virtual memory allows us to make an application program think that it has more memory than actually exists. There are two reasons why one would want this: the first is to allow the use of programs that are too big to physically fit in memory. The other reason is to allow for multitasking – multiple programs running at once. Before virtual memory existed, a word processor, e-mail program, and browser couldn’t be run at the same time unless there was enough memory to hold all three programs at once. This would mean that one would have to close one program in order to run the other, but now with virtual memory, multitasking is possible even when there is not enough memory to hold all executing programs at once.

Virtual Memory Can Slow Down Performance

However, virtual memory can slow down performance. If the size of virtual memory is quite large in comparison to the real memory, then more swapping to and from the hard disk will occur as a result. Accessing the hard disk is far slower than using system memory. Using too many programs at once in a system with an insufficient amount of RAM results in constant disk swapping – also called thrashing, which can really slow down a system’s performance.

 Nice article - http://www.programmerinterview.com/index.php/operating-systems/how-virtual-memory-works/

NASA Public Lessons Learned System - Loved it.

Click here - http://llis.nasa.gov/ to go through some incidents in NASA and learns learnt


Welcome to the NASA Public Lessons Learned System!
The NASA Lessons Learned system provides access to official, reviewed lessons learned from NASA programs and projects. These lessons have been made available to the public by the NASA Office of the Chief Engineer and the NASA Engineering Network. Each lesson describes the original driving event and provides recommendations that feed into NASA’s continual improvement via training, best practices, policies, and procedures.

I read 5-6 incidents and found that the root causes are very similar to the ones we come across in software world. Yet lots to learn from this site. I wish I can find time and go through all of them, you just become that much more wiser, isn't it?

Thanks to all contributors.

Using stringBuilder used for large objects

Came across a thread dump of a degraded Application server due to high heap utlization. From the face of it realized it was due to one type of report and it fetching lot of records. But they say the devil lies in the detail. If we look at the dump closely we would see following -


at java.util.Arrays.copyOf(Arrays.java:2367)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
at java.lang.StringBuilder.append(StringBuilder.java:132)

The first five lines seem to broadly do the following internally -

1) Initialize a StringBuilder()
Construct a string builder with no characters in it and an initial capacity of 16 characters.

2) Append characters (resulset) to it (In heap the top hogger is character array)

3) Check for capacity by internally - public void ensureCapacity(int minimumCapacity)
Ensures that the capacity is at least equal to the specified minimum. If the current capacity is less than the argument, then a new internal array is allocated with greater capacity. The new capacity is the larger of:

• The minimumCapacity argument.
• Twice the old capacity, plus 2.

If the minimumCapacity argument is nonpositive, this method takes no action and simply returns.
Parameters: minimumCapacity - the minimum desired capacity.

Reference - https://docs.oracle.com/javase/7/docs/api/java/lang/StringBuilder.html

Surprise Element - In the worst case scenario a StringBuilder can use about required memory *3. That's at that point where the buffer is increased. Right after the copying the usage drops to 2x.

Reference - http://kaioa.com/node/59?wb48617274=B43136AF

What can be done ?

1. Set some initial capacity to a stringbuilder to override the default and reduce instances of expandcapacity and TEST

StringBuilder(int capacity)
Constructs a string builder with no characters in it and an initial capacity specified by the capacity argument.

Wait Time Psychology Principles

Wait Time Psychology Principles

Below are the propositions:

1. Unoccupied time feels longer than occupied time.
2. Process-waits feel longer than in-process waits.
3. Anxiety makes waits seem longer.
4. Uncertain waits seem longer than known, finite waits.
5. Unfair waits are longer than equitable waits.
6. The more valuable the service, the longer the customer is willing to wait.
7. Solo waits feel longer than group waits.