Definitions of RPO and RTO

RPO (Recovery Point Objective) refers to the amount of data at risk. It's determined by the amount of time between data protection events and reflects the amount of data that potentially could be lost during a disaster recovery. The metric is an indication of the amount of data at risk of being lost.

What does it cost for you to lose 1 hour, 4 hours, 8 hours, 1 day of data? (This is the RPO)

The cost of data loss may be more important than the availability of the application. You may want to invest more in the solution to lower the RPO then to invest in the RTO. Maybe data doesn't change much over at the course of a day, so the solution here may be more simplistic. Again, the answer is not as simple as saying "no data loss."

RTO (Recovery Time Objective) is related to downtime. The metric refers to the amount of time it takes to recover from a data loss event and how long it takes to return to service. RTO refers then to the amount of time the system's data is unavailable or inaccessible preventing normal service.

What does it cost for your application to be unavailable for a 1 hour, 4 hours, 8 hours, 1 day? (This is the RTO)
How can you know what is a good price to pay for a DR solution if you don't know how much it costs you to be down? DR is an insurance policy for your application. You need to know how much to pay to protect the investment (your application). It's not as simple as saying "no downtime."

Linux Memory Usage

Linux free -m

The most common way you'll see on the web to check for free memory in Linux is by using the free command.
Using the free -m command to check your Linux memory usage, displays the values as MB instead of KB.

root@server [~]# free -m
               total    used    free  shared  buffers cached
Mem:            1024    1022     1        0      0     822
-/+ buffers/cache:       200   823
Swap:              0       0      0

Most people will run this command and panic thinking they only have 1 MB of free memory on the server:

The free column beside -/+ buffers/cache with 823 MB is the actual free memory available to Linux.
1024 MB is the total system memory available, which would be physical RAM.
1 MB and 823 MB both show free because an application has access to both for memory storage.
1 MB free plus the 822 MB cached gives the 823 MB of memory actually free to use if needed.

High bit rate video delivery (Non-streaming)

In case of video delivery, especially where it isn't streamed its important to keep an eye on the bit rate which governs the file size and hence the download time before which the user would be able to see the video.

For a video length of say 3 mins following would be the size for different fixed bit rates - 1.25 and 5 Mbps -

1.25 Mbps * 3 * 60 =  225/8 = 28 MB
5 Mbps * 3 * 60 = 900/8 = 112.5 MB

File size = bitrate x duration

What is virtual memory, how is it implemented, and why do operating systems use it?

Real, or physical, memory exists on RAM chips inside the computer. Virtual memory, as its name suggests, doesn’t physically exist on a memory chip. It is an optimization technique and is implemented by the operating system in order to give an application program the impression that it has more memory than actually exists. Virtual memory is implemented by various operating systems such as Windows, Mac OS X, and Linux.
So how does virtual memory work? Let’s say that an operating system needs 120 MB of memory in order to hold all the running programs, but there’s currently only 50 MB of available physical memory stored on the RAM chips. The operating system will then set up 120 MB of virtual memory, and will use a program called the virtual memory manager (VMM) to manage that 120 MB. The VMM will create a file on the hard disk that is 70 MB (120 – 50) in size to account for the extra memory that’s needed. The O.S. will now proceed to address memory as if there were actually 120 MB of real memory stored on the RAM, even though there’s really only 50 MB. So, to the O.S., it now appears as if the full 120 MB actually exists. It is the responsibility of the VMM to deal with the fact that there is only 50 MB of real memory.

The paging file and the RAM

Now, how does the VMM function? As mentioned before, the VMM creates a file on the hard disk that holds the extra memory that is needed by the O.S., which in our case is 70 MB in size. This file is called a paging file (also known as a swap file), and plays an important role in virtual memory. The paging file combined with the RAM accounts for all of the memory. Whenever the O.S. needs a ‘block’ of memory that’s not in the real (RAM) memory, the VMM takes a block from the real memory that hasn’t been used recently, writes it to the paging file, and then reads the block of memory that the O.S. needs from the paging file. The VMM then takes the block of memory from the paging file, and moves it into the real memory – in place of the old block. This process is called swapping (also known as paging), and the blocks of memory that are swapped are called pages. The group of pages that currently exist in RAM, and that are dedicated to a specific process, is known as the working set for that process.
As mentioned earlier, virtual memory allows us to make an application program think that it has more memory than actually exists. There are two reasons why one would want this: the first is to allow the use of programs that are too big to physically fit in memory. The other reason is to allow for multitasking – multiple programs running at once. Before virtual memory existed, a word processor, e-mail program, and browser couldn’t be run at the same time unless there was enough memory to hold all three programs at once. This would mean that one would have to close one program in order to run the other, but now with virtual memory, multitasking is possible even when there is not enough memory to hold all executing programs at once.

Virtual Memory Can Slow Down Performance

However, virtual memory can slow down performance. If the size of virtual memory is quite large in comparison to the real memory, then more swapping to and from the hard disk will occur as a result. Accessing the hard disk is far slower than using system memory. Using too many programs at once in a system with an insufficient amount of RAM results in constant disk swapping – also called thrashing, which can really slow down a system’s performance.

 Nice article - http://www.programmerinterview.com/index.php/operating-systems/how-virtual-memory-works/

NASA Public Lessons Learned System - Loved it.

Click here - http://llis.nasa.gov/ to go through some incidents in NASA and learns learnt


Welcome to the NASA Public Lessons Learned System!
The NASA Lessons Learned system provides access to official, reviewed lessons learned from NASA programs and projects. These lessons have been made available to the public by the NASA Office of the Chief Engineer and the NASA Engineering Network. Each lesson describes the original driving event and provides recommendations that feed into NASA’s continual improvement via training, best practices, policies, and procedures.

I read 5-6 incidents and found that the root causes are very similar to the ones we come across in software world. Yet lots to learn from this site. I wish I can find time and go through all of them, you just become that much more wiser, isn't it?

Thanks to all contributors.

Using stringBuilder used for large objects

Came across a thread dump of a degraded Application server due to high heap utlization. From the face of it realized it was due to one type of report and it fetching lot of records. But they say the devil lies in the detail. If we look at the dump closely we would see following -


at java.util.Arrays.copyOf(Arrays.java:2367)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
at java.lang.StringBuilder.append(StringBuilder.java:132)

The first five lines seem to broadly do the following internally -

1) Initialize a StringBuilder()
Construct a string builder with no characters in it and an initial capacity of 16 characters.

2) Append characters (resulset) to it (In heap the top hogger is character array)

3) Check for capacity by internally - public void ensureCapacity(int minimumCapacity)
Ensures that the capacity is at least equal to the specified minimum. If the current capacity is less than the argument, then a new internal array is allocated with greater capacity. The new capacity is the larger of:

• The minimumCapacity argument.
• Twice the old capacity, plus 2.

If the minimumCapacity argument is nonpositive, this method takes no action and simply returns.
Parameters: minimumCapacity - the minimum desired capacity.

Reference - https://docs.oracle.com/javase/7/docs/api/java/lang/StringBuilder.html

Surprise Element - In the worst case scenario a StringBuilder can use about required memory *3. That's at that point where the buffer is increased. Right after the copying the usage drops to 2x.

Reference - http://kaioa.com/node/59?wb48617274=B43136AF

What can be done ?

1. Set some initial capacity to a stringbuilder to override the default and reduce instances of expandcapacity and TEST

StringBuilder(int capacity)
Constructs a string builder with no characters in it and an initial capacity specified by the capacity argument.