Bug 57831

Yes, shouldn't be that hard to add either. If you want to work on that I can provide some support and help on IRC :)

Any way to wrap around android native encoder? like currently wrap around  MediaCodec API for decoders...

Also what about gst-omx(http://cgit.freedesktop.org/gstreamer/gst-omx)? It was used to provide hardware accel decoders(though, OMX can also provide hardware accel encoders) under android in the early days, but didn't enter the SDK.(in favor of MediaCodecs ?)

(In reply to comment #2)
> Any way to wrap around android native encoder? like currently wrap around 
> MediaCodec API for decoders...

Yes, just do the same ;) Basically need to copy gstamcvideodec.c to gstamcvideoenc.c, change base class to GstVideoEncoder and switch everything to compressed input and uncompressed output. Shouldn't take more than a day of work.

> Also what about gst-omx(http://cgit.freedesktop.org/gstreamer/gst-omx)? It
> was used to provide hardware accel decoders(though, OMX can also provide
> hardware accel encoders) under android in the early days, but didn't enter
> the SDK.(in favor of MediaCodecs ?)

OpenMAX IL is not public API on Android.

Created attachment 77369 [details] [review]
[unworkable]A primary patch adding video encoding support

Note: I haven't tested it as the device at hand has no video encoder registered in gst. I'm also not familiar with the detailed logic, so it's probably broken.

The patch is here for review and further debug purposes.

I tried to use it on a pipeline, but was I was only able to produce uncompressed raw video data with it.
I think the main problem with it is that it does not seem to use or to find a codec ....
But I'am not sure of the way how to use it, is there any special parameters to provide to use specific codec ?
Dear Sebastian could you provide us some support on this ?
I would be really happy to help and make it work.

Thank you,
best regards,
Nicolas

I tried another 4.1.1 table, unfortunately I encountered "No supported color formats for video codec" again. Following the code, I find the CodecCapabilities object returned by "getCapabilitiesForType" has zero colorFormats.

It reports a series of "OMX.rk..." codec, seems be wrappen around HW video func of the underlying rockchip.

I tried to work around this by specifying a color format[1], tried each color format with viddec, none of them worked.

So, the MediaCodec APIs are not properly implemented widely? At least for my case.

BTW, what's the difference between amcviddec-omxrkvideodecoderXXX and amcviddec-XXXdecoder?

--------
[1] Attempt to work around(but failed) 0 colorFormats by specifying a color format
--- a/sys/androidmedia/gstamc.c
+++ b/sys/androidmedia/gstamc.c
@@ -1848,9 +1848,23 @@ scan_codecs (GstPlugin * plugin)
 
       if (g_str_has_prefix (gst_codec_type->mime, "video/")) {
         if (!n_elems) {
+#if 1
+          GST_WARNING ("No supported color formats for video codec, assume COLOR_TI_FormatYUV420PackedSemiPlana
+          n_elems = 1;
+          gst_codec_type->n_color_formats = 1;
+          gst_codec_type->color_formats = g_new0 (gint, 1);
+
+          gst_codec_type->color_formats[0] = COLOR_TI_FormatYUV420PackedSemiPlanar;
+          /*
+           * COLOR_FormatYUV420Planar 
+           * COLOR_TI_FormatYUV420PackedSemiPlanar(COLOR_TI_FormatYUV420PackedSemiPlanarInterlaced)
+           * COLOR_QCOM_FormatYUV420SemiPlanar(COLOR_FormatYUV420SemiPlanar)
+           */
+#else
           GST_ERROR ("No supported color formats for video codec");
           valid_codec = FALSE;
           goto next_supported_type;
+#endif
         }
 
         if (!ignore_unknown_color_formats

(In reply to comment #6)
> I tried another 4.1.1 table, unfortunately I encountered "No supported color
> formats for video codec" again. Following the code, I find the
> CodecCapabilities object returned by "getCapabilitiesForType" has zero
> colorFormats.
> 
> It reports a series of "OMX.rk..." codec, seems be wrappen around HW video
> func of the underlying rockchip.
> 
> I tried to work around this by specifying a color format[1], tried each
> color format with viddec, none of them worked.
> 
> So, the MediaCodec APIs are not properly implemented widely? At least for my
> case.

Looks like it, to be sure you could create a small Java application for Android and check if it returns 0 there too. This is clearly a bug in the implementation of the MediaCodec API of your device.

> BTW, what's the difference between amcviddec-omxrkvideodecoderXXX and
> amcviddec-XXXdecoder?

They're just wrapper plugins for different codecs of the system. Take a look at /etc/media-codecs.xml for the list of all of them.

(In reply to comment #6)
> I tried another 4.1.1 table, unfortunately I encountered "No supported color
> formats for video codec" again. Following the code, I find the
> CodecCapabilities object returned by "getCapabilitiesForType" has zero
> colorFormats.
> 
> It reports a series of "OMX.rk..." codec, seems be wrappen around HW video
> func of the underlying rockchip.
> 
> I tried to work around this by specifying a color format[1], tried each
> color format with viddec, none of them worked.
> 
> So, the MediaCodec APIs are not properly implemented widely? At least for my
> case.
> 

Yes it is the same for me, it seems that every video codec throws a warning saying "Unknown color format 0x00000000".

(In reply to comment #8)
> Yes it is the same for me, it seems that every video codec throws a warning
> saying "Unknown color format 0x00000000".
The returned color format 0x00000000 seems invalid. Here is a list of valid color format codes: http://developer.android.com/reference/android/media/MediaCodecInfo.CodecCapabilities.html#COLOR_Format12bitRGB444

I tried another android 4.1 table, amc found a series of HW-accel codecs with color format code "25"(COLOR_FormatYCbYCr). I tried to map COLOR_FormatYCbYCr to GST_VIDEO_FORMAT_IYU1, it worked for amcvideodec-omxk3videodecoderavc(though the color is not correct), but failed for amcvideodec-omxk3videodecodervp8:
D/GStreamer+amc(15616): 0:00:06.722159000 0x5c6aa600 gstamc.c:118:gst_amc_attach_current_thread Attaching thread 0x5c6aa600
E/OmxVp8DecComponent( 1747): setParameter line 242 ret = 80001005
W/ACodec  (15616): [OMX.k3.video.decoder.vp8] Failed to set standard component role 'video_decoder.vpx'.
E/ACodec  (15616): [OMX.k3.video.decoder.vp8] configureCodec returning error -2147483648
E/MediaCodec(15616): Codec reported an error. (omx error 0x80001001, internalError -2147483648)
E/GStreamer+amc(15616): 0:00:06.739552000 0x5c6aa600 gstamc.c:302:gst_amc_codec_configure Failed to call Java method

BTW, I also tried amcvideodec-omxgooglevpxdecoder and amcviddec-omxgoogleh264decoder, they failed too:
E/OMXNodeInstance(13696): OMX_GetExtensionIndex failed
E/SoftVPX (13696): on2 decoder failed to decode frame.
D/GStreamer+amc(13696): 0:00:02.042483000 0x5c79d6c0 gstamc.c:118:gst_amc_attach_current_thread Attaching thread 0x5c79d6c0
E/ACodec  (13696): [OMX.google.vpx.decoder] ERROR(0x80001001)
E/MediaCodec(13696): Codec reported an error. (omx error 0x80001001, internalError -2147483648)
----
E/OMXNodeInstance(22714): OMX_GetExtensionIndex failed
E/SoftAVC (22714): Decoder failed: -2
E/ACodec  (22714): [OMX.google.h264.decoder] ERROR(0x80001001)
E/MediaCodec(22714): Codec reported an error. (omx error 0x80001001, internalError -2147483648)

Some questions:
* In the failure case, I've got a 0x80001001 ERROR, what does that suggest?
* OMX.google.h264.decoder and OMX.google.vpx.decoder are soft decoding implementations, they should work, unless something wrong with the invocation steps, right?
* For amcvideodec-omxk3videodecoderavc, I saw the following warnings, any ideas?
a) gstamcvideodec.c:1026:gst_amc_video_dec_loop:<amcvideodec-omxk3videodecoderavc0> Frame is too late, dropping (deadline 0:00:00.003577385)
b) I/OmxAvcDecComponent( 1747): in thumbnailmode copy yuv out

(In reply to comment #9)
> (In reply to comment #8)
> > Yes it is the same for me, it seems that every video codec throws a warning
> > saying "Unknown color format 0x00000000".
> The returned color format 0x00000000 seems invalid. Here is a list of valid
> color format codes:
> http://developer.android.com/reference/android/media/MediaCodecInfo.
> CodecCapabilities.html#COLOR_Format12bitRGB444
> 
> I tried another android 4.1 table, amc found a series of HW-accel codecs
> with color format code "25"(COLOR_FormatYCbYCr). I tried to map
> COLOR_FormatYCbYCr to GST_VIDEO_FORMAT_IYU1

YCbYCr is GST_VIDEO_FORMAT_YUY2

> it worked for
> amcvideodec-omxk3videodecoderavc(though the color is not correct), but
> failed for amcvideodec-omxk3videodecodervp8:
> D/GStreamer+amc(15616): 0:00:06.722159000 0x5c6aa600
> gstamc.c:118:gst_amc_attach_current_thread Attaching thread 0x5c6aa600
> E/OmxVp8DecComponent( 1747): setParameter line 242 ret = 80001005
> W/ACodec  (15616): [OMX.k3.video.decoder.vp8] Failed to set standard
> component role 'video_decoder.vpx'.
> E/ACodec  (15616): [OMX.k3.video.decoder.vp8] configureCodec returning error
> -2147483648
> E/MediaCodec(15616): Codec reported an error. (omx error 0x80001001,
> internalError -2147483648)

That's OMX_ErrorUndefined, useful :)

> E/GStreamer+amc(15616): 0:00:06.739552000 0x5c6aa600
> gstamc.c:302:gst_amc_codec_configure Failed to call Java method
> 
> BTW, I also tried amcvideodec-omxgooglevpxdecoder and
> amcviddec-omxgoogleh264decoder, they failed too:
> E/OMXNodeInstance(13696): OMX_GetExtensionIndex failed
> E/SoftVPX (13696): on2 decoder failed to decode frame.
> D/GStreamer+amc(13696): 0:00:02.042483000 0x5c79d6c0
> gstamc.c:118:gst_amc_attach_current_thread Attaching thread 0x5c79d6c0
> E/ACodec  (13696): [OMX.google.vpx.decoder] ERROR(0x80001001)
> E/MediaCodec(13696): Codec reported an error. (omx error 0x80001001,
> internalError -2147483648)
> ----
> E/OMXNodeInstance(22714): OMX_GetExtensionIndex failed
> E/SoftAVC (22714): Decoder failed: -2
> E/ACodec  (22714): [OMX.google.h264.decoder] ERROR(0x80001001)
> E/MediaCodec(22714): Codec reported an error. (omx error 0x80001001,
> internalError -2147483648)
> 
> Some questions:
> * In the failure case, I've got a 0x80001001 ERROR, what does that suggest?

That someone was too lazy to add a useful error value there and do proper error reporting.

> * OMX.google.h264.decoder and OMX.google.vpx.decoder are soft decoding
> implementations, they should work, unless something wrong with the
> invocation steps, right?

Yes

> * For amcvideodec-omxk3videodecoderavc, I saw the following warnings, any
> ideas?
> a)
> gstamcvideodec.c:1026:gst_amc_video_dec_loop:<amcvideodec-
> omxk3videodecoderavc0> Frame is too late, dropping (deadline
> 0:00:00.003577385)
> b) I/OmxAvcDecComponent( 1747): in thumbnailmode copy yuv out

a) happens if a frame is decoded too late and would be displayed too late

b) should not be a problem

Created attachment 80833 [details] [review]
A primary patch(against sdk 2012.11) adding video encoding support

This patch works for my Huawei MediaPad 10 FHD.
The testing pipeline:
"videotestsrc ! capsfilter caps=\"video/x-raw-yuv,format=(fourcc)NV12,framerate=(fraction)8/1\" ! amcvidenc-omxk3videoencoderavc bitrate=1024 i-frame-int=5 ! rtph264pay ! queue ! udpsink host=192.168.1.100 port=3057"

The underlying hardware reports three supported color formats:
* COLOR_FormatYUV420SemiPlanar (GST_VIDEO_FORMAT_NV12)
* COLOR_FormatYCbYCr (GST_VIDEO_FORMAT_YUY2)
Tested with "videotestsrc ! video/x-raw-yuv,format=(fourcc)YUY2...", and got a "gst_amc_video_enc_handle_frame: error: Failed to queue input buffer"
* COLOR_Format32bitARGB8888 (GST_VIDEO_FORMAT_ARGB)
Tested with "videotestsrc ! video/x-raw-rgb,bpp=32,depth=32,endianness=4321,alpha_mask=-16777216,red_mask=16711680,green_mask=65280,blue_mask=255,...", and got the same error.

Any idea? It seems there is no stride/crop concept for amc encoder...

Please Review:
1. _find_nearest_frame()
* Is it necessary for encoder? 
* The commented out "gst_video_enoder_drop_frame ...", there is no drop_frame for encoder, any replacement?

2. gst_amc_video_enc_handle_output_frame()
* What do the different branches take care of?

3. gst_amc_video_enc_drain()
* Copy and modify from the "dec" version, need more modification?

= TODO =
No "profile" setting for encoder? At least according to the document of MediaFormat.

----
For references:
* http://developer.android.com/reference/android/media/MediaCodec.html
* http://developer.android.com/reference/android/media/MediaCodecInfo.CodecCapabilities.html
* http://developer.android.com/reference/android/media/MediaFormat.html
* Example1: https://gist.github.com/wobbals/3990442
* Example2: https://android.googlesource.com/platform/cts/+/6289d680cb75fa5a985464b9f362d4a2a007a7bf%5E!/#F0

Created attachment 80894 [details] [review]
Patch(against sdk 2012.11, Version 2) adding video encoding support

Changes since v1:
* gst_amc_video_enc_set_format()
--> Negotiating the src caps (Right now there's no src pad caps negotiation for amcvideo**dec**?)
--> Call gst_amc_video_enc_set_src_caps if !encoder->started (For encoder, it seems will not return INFO_OUTPUT_FORMAT_CHANGED the first time)

Fixups for raw buffer filling, etc.

Created attachment 81539 [details] [review]
Patch(against sdk 2012.11, Version 3) adding video encoding support

Version 3 of the patch against sdk 2012.11, changes since V2:
1. Remove mappings of COLOR_FormatYCbYCr & COLOR_Format32bitARGB8888
2. Set src pad's caps in gst_amc_video_enc_loop:
* On the start of codec, self->first_set_format will be assigned with GstAmcFormat that is passed in gst_amc_codec_configure
* In gst_amc_video_enc_loop, it will set src pad's GstVideoCodecState if self->first_set_format is set or INFO_OUTPUT_FORMAT_CHANGED; self->first_set_format will then be cleared.
3. Convert raw data to respect to "stride > width" case, that's processed in gst_amc_video_enc_fill_buffer.
4. More fileds are set in GstAmcFormat while configring the codec:
"stride": width be rounded up to a multiple of 4 (the same as gst-omx's "safe" logic.)
"slice-height": equal to height
"profile & level": Not set currently. If set "profile" to any value of AVCProfile*, codec configuration failed.
5. Handle incorrectly reported color, that's in color_format_to_video_format_hack & video_format_to_color_format_hack
6. Bug fix & code cleanup