In h264encode, we observe that memory usage increases as the program proceeds.  For large files, this results in unreasonably large memory usage.

Example case:

Input: 18G yuv file
Output: 3.3G h264 video
run with valgrind using the massif tool:
$ valgrind --tool=massif --stacks=yes /usr/local/bin/h264encode --srcyuv output2.yuv -framecount 0 -f 25 -o massif_test_out.mp4

There is a malloc at i965_drv_video.c:1671 that accounts for 92.58% of memory usage at the peak.  This call creates a buffer which has a reference count on it.  When the reference count hits 0, the memory is cleared.  Many of the buffers stored in this way never have their reference counts set to zero during encoding.  As a result, the memory usage keeps growing during encoding.  On completion, all of this memory is freed, so a simple valgrind run does not find this leak.

Excerpts from the ms_print massif file:
--------------------------------------------------------------------------------
Command:            /usr/local/bin/h264encode --srcyuv output2.yuv -framecount 0 -f 25 -o massif_test_out
Massif arguments:   --stacks=yes
ms_print arguments: massif.out.31611
--------------------------------------------------------------------------------

    GB
1.202^                                                                       #
     |                                                                   @@@@#
     |                                                                @@@@@@@#
     |                                                             @@@@@@@@@@#
     |                                                          @@@@@@@@@@@@@#
     |                                                       @@@@@@@@@@@@@@@@#
     |                                                    :@@@ @@@@@@@@@@@@@@#
     |                                                @:@@:@ @ @@@@@@@@@@@@@@#
     |                                            @@@@@:@@:@ @ @@@@@@@@@@@@@@#
     |                                        @@@@@@@@@:@@:@ @ @@@@@@@@@@@@@@#
     |                                     :::@@@ @@@@@:@@:@ @ @@@@@@@@@@@@@@#
     |                                 @:::: :@@@ @@@@@:@@:@ @ @@@@@@@@@@@@@@#
     |                              @@@@:::: :@@@ @@@@@:@@:@ @ @@@@@@@@@@@@@@#
     |                         @@:@@@@ @:::: :@@@ @@@@@:@@:@ @ @@@@@@@@@@@@@@#
     |                      @@@@@:@ @@ @:::: :@@@ @@@@@:@@:@ @ @@@@@@@@@@@@@@#
     |                   :@@@@@@@:@ @@ @:::: :@@@ @@@@@:@@:@ @ @@@@@@@@@@@@@@#
     |                @@::@ @@@@@:@ @@ @:::: :@@@ @@@@@:@@:@ @ @@@@@@@@@@@@@@#
     |           @@:::@ ::@ @@@@@:@ @@ @:::: :@@@ @@@@@:@@:@ @ @@@@@@@@@@@@@@#
     |       @@@:@@:: @ ::@ @@@@@:@ @@ @:::: :@@@ @@@@@:@@:@ @ @@@@@@@@@@@@@@#
     |    @@:@@ :@@:: @ ::@ @@@@@:@ @@ @:::: :@@@ @@@@@:@@:@ @ @@@@@@@@@@@@@@#
   0 +----------------------------------------------------------------------->Gi
     0                                                                   75.59

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------

 79 80,394,261,652    1,290,739,176    1,255,405,747    35,332,005        1,424
97.26% (1,255,405,747B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->92.58% (1,194,924,096B) 0x567936F: i965_create_buffer_internal.isra.18 (i965_drv_video.c:1671)
| ->92.58% (1,194,924,096B) 0x4C27E60: vaCreateBuffer (va.c:949)
|   ->78.39% (1,011,857,700B) 0x4026E5: main (h264encode.c:1656)
|   | 
|   ->13.44% (173,461,320B) 0x40664B: render_picture (h264encode.c:1426)
|   | ->13.21% (170,570,160B) 0x402634: main (h264encode.c:1995)
|   | | 
|   | ->00.22% (2,891,160B) in 1+ places, all below ms_print's threshold (01.00%)
|   | 
|   ->00.74% (9,605,076B) in 1+ places, all below ms_print's threshold (01.00%)
|   
->02.51% (32,397,504B) 0x569361B: object_heap_expand (object_heap.c:62)
| ->02.51% (32,382,464B) 0x56937FE: object_heap_allocate (object_heap.c:121)
| | ->02.51% (32,379,904B) 0x5679235: i965_create_buffer_internal.isra.18 (i965_drv_video.c:1612)
| | | ->02.38% (30,683,648B) 0x567C0CB: i965_DeriveImage (i965_drv_video.c:3041)
| | | | ->02.38% (30,683,648B) 0x405464: load_surface (loadsurface.h:304)
| | | |   ->02.38% (30,683,136B) 0x405F1E: storage_task (h264encode.c:1903)
| | | |   | ->02.38% (30,683,136B) 0x4061A7: storage_task_thread (h264encode.c:1925)
| | | |   |   ->02.38% (30,683,136B) 0x3A95607C51: start_thread (in /usr/lib64/libpthread-2.17.so)
| | | |   |     ->02.38% (30,683,136B) 0x3A94EF5E1B: clone (in /usr/lib64/libc-2.17.so)
| | | |   |       
| | | |   ->00.00% (512B) in 1+ places, all below ms_print's threshold (01.00%)
| | | |   
| | | ->00.13% (1,696,256B) in 1+ places, all below ms_print's threshold (01.00%)
| | | 
| | ->00.00% (2,560B) in 1+ places, all below ms_print's threshold (01.00%)
| | 
| ->00.00% (15,040B) in 1+ places, all below ms_print's threshold (01.00%)
| 
->01.88% (24,285,024B) 0x5679288: i965_create_buffer_internal.isra.18 (i965_drv_video.c:1629)
| ->01.88% (24,285,024B) 0x4C27E60: vaCreateBuffer (va.c:949)
| | ->01.88% (24,285,024B) in 9 places, all below massif's threshold (01.00%)
| |   
| ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| 
->00.29% (3,799,123B) in 1+ places, all below ms_print's threshold (01.00%)