i.mx8의 selfie_segmentrer.py 에서 발췌
# Set backend and delegates
if self.backend == "CPU":
if self.platform == "i.MX8MP":
backend = "true:CPU custom=NumThreads:4"
else:
backend = "true:CPU custom=NumThreads:2"
else:
if self.platform == "i.MX8MP":
os.environ["USE_GPU_INFERENCE"] = "0"
backend = (
"true:npu custom=Delegate:External,ExtDelegateLib:libvx_delegate.so"
)
else:
backend = "true:npu custom=Delegate:External,ExtDelegateLib:libethosu_delegate.so"
|
+ " ! videoconvert ! video/x-raw,format=RGB ! tensor_converter ! "
+ "tensor_transform mode=arithmetic option=typecast:float32,div:255.0 ! "
+ "tensor_filter framework=tensorflow-lite model="
+ self.tflite_model
+ " accelerator="
+ backend
+ " name=tensor_filter latency=1 ! tensor_sink name=tensor_sink " |
gsteramer의 엘리먼트를 기본으로 NN에 맞춰서 몇개 추가한 것 같은데
[링크 : https://nnstreamer.github.io/gst/nnstreamer/tensor_converter/README.html]
[링크 : https://nnstreamer.github.io/gst/nnstreamer/tensor_transform/README.html]
[링크 : https://nnstreamer.github.io/gst/nnstreamer/tensor_decoder/README.html]
[링크 : https://nnstreamer.github.io/gst/nnstreamer/tensor_filter/README.html]
[링크 : https://nnstreamer.github.io/gst/nnstreamer/tensor_sink/README.html]
[링크 : https://nnstreamer.github.io/gst/nnstreamer/elements/gsttensor_sink.html]
사용예에서 보면 converter / transform / filter / sink 순으로 쓰는 듯
[CAM] - [videoconvert] - [videoscale] - [tee] -+- [queue] - [videoconvert] - [cairooverlay] - [ximagesink]
+- [queue] - [videoscale] - [tensor_converter] - [tensor_transform] - [tensor_filter] - [tensor_sink] |
[링크 : https://nnstreamer.github.io/how-to-run-examples.html]
+
gst-inspector tensor_converter
더보기
root@imx8mpevk:~# gst-inspect-1.0 tensor_converter
Factory Details:
Rank none (0)
Long-name TensorConverter
Klass Converter/Tensor
Description Converts an audio, video, text, or arbitrary stream to a tensor stream of C-Array for neural network framework filters
Author MyungJoo Ham <myungjoo.ham@samsung.com></myungjoo.ham@samsung.com>
Plugin Details:
Name nnstreamer
Description nnstreamer plugin library
Filename /usr/lib/gstreamer-1.0/libnnstreamer.so
Version 2.4.0
License LGPL
Source module nnstreamer
Binary package nnstreamer
Origin URL https://github.com/nnstreamer/nnstreamer
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstTensorConverter
Pad Templates:
SINK template: 'sink'
Availability: Always
Capabilities:
video/x-raw
format: { (string)RGB, (string)BGR, (string)RGBx, (string)BGRx, (string)xRGB, (string)xBGR, (string)RGBA, (string)BGRA, (string)ARGB, (string)ABGR, (string)GRAY8, (string)GRAY16_BE, (string)GRAY16_LE }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
interlace-mode: progressive
audio/x-raw
format: { (string)S8, (string)U8, (string)S16LE, (string)S16BE, (string)U16LE, (string)U16BE, (string)S32LE, (string)S32BE, (string)U32LE, (string)U32BE, (string)F32LE, (string)F32BE, (string)F64LE, (string)F64BE }
rate: [ 1, 2147483647 ]
channels: [ 1, 2147483647 ]
layout: interleaved
text/x-raw
format: utf8
application/octet-stream
other/tensors
format: flexible
framerate: [ 0/1, 2147483647/1 ]
application/octet-stream
other/protobuf-tensor
framerate: [ 0/1, 2147483647/1 ]
SRC template: 'src'
Availability: Always
Capabilities:
other/tensor
framerate: [ 0/1, 2147483647/1 ]
other/tensors
format: { (string)static, (string)flexible }
framerate: [ 0/1, 2147483647/1 ]
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SINK: 'sink'
Pad Template: 'sink'
SRC: 'src'
Pad Template: 'src'
Element Properties:
frames-per-tensor : The number of frames in output tensor
flags: readable, writable
Unsigned Integer. Range: 1 - 4294967295 Default: 1
input-dim : Input tensor dimension from inner array
flags: readable, writable
String. Default: ""
input-type : Type of each element of the input tensor
flags: readable, writable
String. Default: ""
mode : Converter mode. e.g., mode=custom-code:. For detail, refer to </registered callback name>https://github.com/nnstreamer/nnstreamer/blob/main/gst/nnstreamer/elements/gsttensor_converter.md#custom-converter
flags: readable, writable
String. Default: ""
name : The name of the object
flags: readable, writable
String. Default: "tensorconverter0"
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
set-timestamp : The flag to set timestamp when received a buffer with invalid timestamp
flags: readable, writable
Boolean. Default: true
silent : Produce verbose output
flags: readable, writable
Boolean. Default: true
sub-plugins : Registrable sub-plugins list
flags: readable
String. Default: "python3,protobuf" |
gst-inspector tensor_transform
더보기
root@imx8mpevk:~# gst-inspect-1.0 tensor_transform
Factory Details:
Rank none (0)
Long-name TensorTransform
Klass Filter/Tensor
Description Transforms other/tensor dimensions for different models or frameworks
Author MyungJoo Ham <myungjoo.ham@samsung.com>
Plugin Details:
Name nnstreamer
Description nnstreamer plugin library
Filename /usr/lib/gstreamer-1.0/libnnstreamer.so
Version 2.4.0
License LGPL
Source module nnstreamer
Binary package nnstreamer
Origin URL https://github.com/nnstreamer/nnstreamer
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstBaseTransform
+----GstTensorTransform
Pad Templates:
SINK template: 'sink'
Availability: Always
Capabilities:
other/tensor
framerate: [ 0/1, 2147483647/1 ]
other/tensors
format: { (string)static, (string)flexible }
framerate: [ 0/1, 2147483647/1 ]
SRC template: 'src'
Availability: Always
Capabilities:
other/tensor
framerate: [ 0/1, 2147483647/1 ]
other/tensors
format: { (string)static, (string)flexible }
framerate: [ 0/1, 2147483647/1 ]
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SINK: 'sink'
Pad Template: 'sink'
SRC: 'src'
Pad Template: 'src'
Element Properties:
acceleration : Orc acceleration
flags: readable, writable
Boolean. Default: true
apply : Select tensors to apply, separated with ',' in case of multiple tensors. Default to apply all tensors.
flags: readable, writable
String. Default: ""
mode : Mode used for transforming tensor
flags: readable, writable
Enum "gtt_mode_type" Default: -1, "unknown"
(0): dimchg - Mode for changing tensor dimensions, option=FROM_DIM:TO_DIM (with a regex, ^([0-9]|1[0-5]):([0-9]|1[0-5])$, where NNS_TENSOR_RANK_LIMIT is 16)
(1): typecast - Mode for casting type of tensor, option=(^[u]?int(8|16|32|64)$|^float(16|32|64)$)
(2): arithmetic - Mode for arithmetic operations with tensor, option=[typecast:TYPE,][per-channel:(false|true@DIM),]add|mul|div:NUMBER[@CH_IDX], ...
(3): transpose - Mode for transposing shape of tensor, option=D1':D2':D3':D4 (fixed to 3)
(4): stand - Mode for statistical standardization of tensor, option=(default|dc-average)[:TYPE][,per-channel:(false|true)]
(5): clamp - Mode for clamping all elements of tensor into the range, option=CLAMP_MIN:CLAMP_MAX
(-1): unknown - Unknown or not-implemented-yet mode
name : The name of the object
flags: readable, writable
String. Default: "tensortransform0"
option : Option for the tensor transform mode ?
flags: readable, writable
String. Default: null
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
qos : Handle Quality-of-Service events
flags: readable, writable
Boolean. Default: false
silent : Produce verbose output ?
flags: readable, writable
Boolean. Default: true
transpose-rank-limit: The rank limit of transpose, which varies per version of nnstreamer and may be lower than the global rank limit if it is over 4.
flags: readable
Unsigned Integer. Range: 0 - 16 Default: 4 |
gst-inspector tensor_decoder
더보기
root@imx8mpevk:~# gst-inspect-1.0 tensor_decoder
Factory Details:
Rank none (0)
Long-name TensorDecoder
Klass Converter/Tensor
Description Converts tensor stream of C-Array for neural network framework filters to audio or video stream
Author Jijoong Moon <jijoong.moon@samsung.com>
Plugin Details:
Name nnstreamer
Description nnstreamer plugin library
Filename /usr/lib/gstreamer-1.0/libnnstreamer.so
Version 2.4.0
License LGPL
Source module nnstreamer
Binary package nnstreamer
Origin URL https://github.com/nnstreamer/nnstreamer
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstBaseTransform
+----GstTensorDecoder
Pad Templates:
SINK template: 'sink'
Availability: Always
Capabilities:
other/tensor
framerate: [ 0/1, 2147483647/1 ]
other/tensors
format: static
num_tensors: [ 1, 16 ]
framerate: [ 0/1, 2147483647/1 ]
other/tensors
format: flexible
framerate: [ 0/1, 2147483647/1 ]
SRC template: 'src'
Availability: Always
Capabilities:
ANY
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SINK: 'sink'
Pad Template: 'sink'
SRC: 'src'
Pad Template: 'src'
Element Properties:
config-file : sets config file path which contains plugins properties
flags:
** (gst-inspect-1.0:1706): WARNING **: 05:54:08.872: /usr/src/debug/nnstreamer/2.4.0/gst/nnstreamer/elements/gsttensor_decoder.c:592: invalid property id 13 for "config-file" of type 'GParamString' in 'GstTensorDecoder'
readable, writable
String. Default: null
mode : Decoder mode
flags: readable, writable
String. Default: ""
name : The name of the object
flags: readable, writable
String. Default: "tensordecoder0"
option1 : option for specific decoder modes, 1st one.
flags: readable, writable
String. Default: null
option2 : option for specific decoder modes, 2nd one.
flags: readable, writable
String. Default: null
option3 : option for specific decoder modes, 3rd one.
flags: readable, writable
String. Default: null
option4 : option for specific decoder modes, 4th one.
flags: readable, writable
String. Default: null
option5 : option for specific decoder modes, 5th one.
flags: readable, writable
String. Default: null
option6 : option for specific decoder modes, 6th one.
flags: readable, writable
String. Default: null
option7 : option for specific decoder modes, 7th one.
flags: readable, writable
String. Default: null
option8 : option for specific decoder modes, 8th one.
flags: readable, writable
String. Default: null
option9 : option for specific decoder modes, 9th one.
flags: readable, writable
String. Default: null
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
qos : Handle Quality-of-Service events
flags: readable, writable
Boolean. Default: false
silent : Produce verbose output
flags: readable, writable
Boolean. Default: true
sub-plugins : Registrable sub-plugins list
flags: readable
String. Default: "protobuf,direct_video,bounding_boxes,image_segment,python3,octet_stream,pose_estimation,tensor_region,image_labeling" |
gst-inspector tensor_filter
더보기
root@imx8mpevk:~# gst-inspect-1.0 tensor_filter
Factory Details:
Rank none (0)
Long-name TensorFilter
Klass Filter/Tensor
Description Handles NN Frameworks (e.g., tensorflow) as Media Filters with other/tensor type stream
Author MyungJoo Ham <myungjoo.ham@samsung.com>
Plugin Details:
Name nnstreamer
Description nnstreamer plugin library
Filename /usr/lib/gstreamer-1.0/libnnstreamer.so
Version 2.4.0
License LGPL
Source module nnstreamer
Binary package nnstreamer
Origin URL https://github.com/nnstreamer/nnstreamer
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstBaseTransform
+----GstTensorFilter
Pad Templates:
SINK template: 'sink'
Availability: Always
Capabilities:
other/tensor
framerate: [ 0/1, 2147483647/1 ]
other/tensors
format: { (string)static, (string)flexible }
framerate: [ 0/1, 2147483647/1 ]
SRC template: 'src'
Availability: Always
Capabilities:
other/tensor
framerate: [ 0/1, 2147483647/1 ]
other/tensors
format: { (string)static, (string)flexible }
framerate: [ 0/1, 2147483647/1 ]
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SINK: 'sink'
Pad Template: 'sink'
SRC: 'src'
Pad Template: 'src'
Element Properties:
accelerator : Set accelerator for the subplugin with format (true/false):(comma separated ACCELERATOR(s)). true/false determines if accelerator is to be used. list of accelerators determines the backend (ignored with false). Example, if GPU, NPU can be used but not CPU - true:npu,gpu,!cpu. The full list of accelerators can be found in nnstreamer_plugin_api_filter.h. Note that only a few subplugins support this property.
flags: readable, writable
String. Default: ""
custom : Custom properties for subplugins ?
flags: readable, writable
String. Default: ""
framework : Neural network framework
flags: readable, writable
String. Default: "auto"
input : Input tensor dimension from inner array, up to 4 dimensions ?
flags: readable, writable
String. Default: ""
input-combination : Select the input tensor(s) to invoke the models
flags: readable, writable
String. Default: ""
inputlayout : Set channel first (NCHW) or channel last layout (NHWC) or None for input data. Layout of the data can be any or NHWC or NCHW or none for now.
flags: readable, writable
String. Default: ""
inputname : The Name of Input Tensor
flags: readable, writable
String. Default: ""
inputranks : The Rank of the Input Tensor, which is separated with ',' in case of multiple Tensors
flags: readable
String. Default: ""
inputtype : Type of each element of the input tensor ?
flags: readable, writable
String. Default: ""
invoke-dynamic : Flexible tensors whose memory size changes can be used asinput and output of the tensor filter. With this option, the output caps is always in the format of flexible tensors.
flags: readable, writable
Boolean. Default: false
is-updatable : Indicate whether a given model to this tensor filter is updatable in runtime. (e.g., with on-device training)
flags: readable, writable
Boolean. Default: false
latency : Turn on performance profiling for the average latency over the recent 10 inferences in microseconds. Currently, this accepts either 0 (OFF) or 1 (ON).
flags: readable, writable
Integer. Range: 0 - 1 Default: -1
latency-report : Report to the pipeline the estimated tensor-filter element latency.
flags: readable, writable
Boolean. Default: false
model : File path to the model file. Separated with ',' in case of multiple model files(like caffe2)
flags: readable, writable
String. Default: ""
name : The name of the object
flags: readable, writable
String. Default: "tensorfilter0"
output : Output tensor dimension from inner array, up to 4 dimensions ?
flags: readable, writable
String. Default: ""
output-combination : Select the output tensor(s) from the input tensor(s) and/or model output
flags: readable, writable
String. Default: ""
outputlayout : Set channel first (NCHW) or channel last layout (NHWC) or None for output data. Layout of the data can be any or NHWC or NCHW or none for now.
flags: readable, writable
String. Default: ""
outputname : The Name of Output Tensor
flags: readable, writable
String. Default: ""
outputranks : The Rank of the Out Tensor, which is separated with ',' in case of multiple Tensors
flags: readable
String. Default: ""
outputtype : Type of each element of the output tensor ?
flags: readable, writable
String. Default: ""
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
qos : Handle Quality-of-Service events
flags: readable, writable
Boolean. Default: false
shared-tensor-filter-key: Multiple element instances of tensor-filter in a pipeline may share a single resource instance if they share the same framework (subplugin) and neural network model. Designate "shared-tensor-filter-key" to declare and share such instances. If it is NULL, it means the model representations is not shared.
flags: readable, writable
String. Default: ""
silent : Produce verbose output
flags: readable, writable
Boolean. Default: true
sub-plugins : Registrable sub-plugins list
flags: readable
String. Default: "custom,custom-easy,cpp,python3,tvm,tensorflow2-lite"
throughput : Turn on performance profiling for the average throughput in the number of outputs per seconds (i.e., FPS), multiplied by 1000 to represent a floating point using an integer. Currently, this accepts either 0 (OFF) or 1 (ON).
flags: readable, writable
Integer. Range: 0 - 1 Default: -1 |
gst-inspector tensor_sink
더보기
root@imx8mpevk:~# gst-inspect-1.0 tensor_sink
Factory Details:
Rank none (0)
Long-name TensorSink
Klass Sink/Tensor
Description Sink element to handle tensor stream
Author Samsung Electronics Co., Ltd.
Plugin Details:
Name nnstreamer
Description nnstreamer plugin library
Filename /usr/lib/gstreamer-1.0/libnnstreamer.so
Version 2.4.0
License LGPL
Source module nnstreamer
Binary package nnstreamer
Origin URL https://github.com/nnstreamer/nnstreamer
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstBaseSink
+----GstTensorSink
Pad Templates:
SINK template: 'sink'
Availability: Always
Capabilities:
other/tensor
framerate: [ 0/1, 2147483647/1 ]
other/tensors
format: { (string)static, (string)flexible, (string)sparse }
framerate: [ 0/1, 2147483647/1 ]
Element has no clocking capabilities.
Element has no URI handling capabilities.
Pads:
SINK: 'sink'
Pad Template: 'sink'
Element Properties:
async : Go asynchronously to PAUSED
flags: readable, writable
Boolean. Default: true
blocksize : Size in bytes to pull per buffer (0 = default)
flags: readable, writable
Unsigned Integer. Range: 0 - 4294967295 Default: 4096
emit-signal : Emit signal for new data, stream start, eos
flags: readable, writable
Boolean. Default: true
enable-last-sample : Enable the last-sample property
flags: readable, writable
Boolean. Default: true
last-sample : The last sample received in the sink
flags: readable
Boxed pointer of type "GstSample"
max-bitrate : The maximum bits per second to render (0 = disabled)
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
max-lateness : Maximum number of nanoseconds that a buffer can be late before it is dropped (-1 unlimited)
flags: readable, writable
Integer64. Range: -1 - 9223372036854775807 Default: -1
name : The name of the object
flags: readable, writable
String. Default: "tensorsink0"
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
processing-deadline : Maximum processing time for a buffer in nanoseconds
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 20000000
qos : Generate Quality-of-Service events upstream
flags: readable, writable
Boolean. Default: true
render-delay : Additional render delay of the sink in nanoseconds
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
signal-rate : New data signals per second (0 for unlimited, max 500)
flags: readable, writable
Unsigned Integer. Range: 0 - 500 Default: 0
silent : Produce verbose output
flags: readable, writable
Boolean. Default: true
stats : Sink Statistics
flags: readable
Boxed pointer of type "GstStructure"
average-rate: 0
dropped: 0
rendered: 0
sync : Sync on the clock
flags: readable, writable
Boolean. Default: false
throttle-time : The time to keep between rendered buffers (0 = disabled)
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
ts-offset : Timestamp offset in nanoseconds
flags: readable, writable
Integer64. Range: -9223372036854775808 - 9223372036854775807 Default: 0
Element Signals:
"eos" : void user_function (GstElement * object,
gpointer user_data);
"stream-start" : void user_function (GstElement * object,
gpointer user_data);
"new-data" : void user_function (GstElement * object,
GstBuffer * arg0,
gpointer user_data); |
