SBC: NVIDIA Nano OS: Jetpack 4.4.1 Ubuntu 18.04 DC: v 4.1.0 Camera: Intel D435i depth camera Vehicle: Traxxas 4WD Slash with brushless ESC and motor

When using a PS4 gamepad for local control of my Donkey Car there is a noticeable lag in both the steering and throttle response of the car during "manage.py drive". I have experienced this lag in both DC v 3.1.5 and v 4.1.0.

I have investigated the lag in both the steering and the throttle response to determine whether it is the ESC or the Nano/PCA9685. I used a servo controller to test the Traxxas ESC and it responds correctly to the PWM range of 1.0 to 2.0 ms without hesitation. On the other hand I connected a PWM display to the throttle servo channel output on the PCA9685 to see what the throttle output was like during "manage.py drive". While using the PS4 gamepad I found that when I pushed the right joystick all the way up vertically, there was a definite delay (lag), sometimes short and sometimes long, before the throttle output PWM moved towards 2.0 ms. I experienced the same issue for reverse. There appears to be a definite issue with the drive loop's response to the PS4 gamepad throttle channel input and the appropriate output from the PCA9685 throttle channel output. To make sure that there was not an issue with the PS4 gamepad communicating with the Nano, I paired and connected the PS4 to an identical Traxxas 4WD Slash running a Nano, but instead of a D435i depth camera it was using a Sainsmart 8 Mp color camera. I found that there was no lag between the steering and throttle outputs of the gamepad joysticks and the steering and throttle responses of the Slash steering servo and ESC motor controller.

Based upon my findings, it appears that there is some issue with the Nano when using the D435i depth camera in place of the usual Rpi or Sainsmart color cameras. I have a spare Sainsmart 8 Mp camera that I can use in place of the D435i depth camera to see if the gamepad to robot steering/throttle control latency is caused by the D435i depth camera being in the drive loop.

By disconnecting the D435i depth camera from the Nano and attaching a Sainsmart 8 Mp color camera to the Nano and updating "myconfig.py" to reflect the camera changeout, I verified that the Nano/Sainsmart combination does not have the steering/throttle latency issue that the Nano/D435i depth camera does. Restoring the Nano/D435i depth camera configuration and "myconfig.py" brought back the gamepad steering and throttle latency as I originally experienced. Even updating/upgrading Ubuntu did not cure the latency problem.

I have been working on the Nano/D435i camera project with a DC member in Australia who is using the same configuration, but is much farther ahead with the project than me. Attempts to contact him through Discord Direct Messages have been to no avail as he has been silent for three weeks so I do not have an experienced user who has performed data collection and training to fall back on.

Comments please.

When the python ~/mycar/train.py --tub <tub folder names comma separated> --model ./models/mypilot.h5 command is ran (any form of it), the epochs start but eventually fail with error W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated. [[{{node PyFunc}}]] .

I'm not able to use my physical joystick with web server, but it works fine when I use the "--js" option, but that disables the web server, according to the docs: "The default web controller may be replaced with a one line change to use a physical joystick part for input ...", where is this line, and how do I change it?

Temporarily fix: go to manage.py and changes:

    if use_joystick or cfg.USE_JOYSTICK_AS_DEFAULT:
         ctr = JoystickController(max_throttle=cfg.JOYSTICK_MAX_THROTTLE,
                                  steering_scale=cfg.JOYSTICK_STEERING_SCALE,
                                  auto_record_on_throttle=cfg.AUTO_RECORD_ON_    THROTTLE)
     else:
         # This web controller will create a web server that is capable
         # of managing steering, throttle, and modes, and more.
         ctr = LocalWebController(use_chaos=use_chaos)
 
     V.add(ctr,
           inputs=['cam/image_array'],
           outputs=['user/angle', 'user/throttle', 'user/mode', 'recording'],
           threaded=True)

to:

    
    ctr = LocalWebController(use_chaos=use_chaos)
    V.add(ctr,
           inputs=['cam/image_array'],
           outputs=['user/angle', 'user/throttle', 'user/mode', 'recording'],
           threaded=True)
     ctr = JoystickController(max_throttle=cfg.JOYSTICK_MAX_THROTTLE,
                                  steering_scale=cfg.JOYSTICK_STEERING_SCALE,
                                  auto_record_on_throttle=cfg.AUTO_RECORD_ON_    THROTTLE)
     V.add(ctr,
           inputs=['cam/image_array'],
           outputs=['user/angle', 'user/throttle', 'user/mode', 'recording'],
           threaded=True)

In that way you add the two controller handlers, but the JoystickController needs to be the last. This is a temporally fix, not a bug fix!

This PR allows you to use Fast AI(pytorch) end to end in donkey car. I have tested training and driving with the simulator.

I have added a new type of model fastai_ so far linear is the only model implemented. More can be added in the future of course. I have tried my best to emulate most of the function that Keras does but there are some idiosyncrasies between the two frameworks.

These normal training and driving commands work. python train.py --tubs data/ --model models/fastai_linear_model.pth --type fastai_linear python manage.py drive --model models/fastai_linear_model.pth --type fastai_linear

I do realise that this PR comes without a heads up or warning so if its rejected that's fine. I do really like this project and the work that has been done, its just I prefer PyTorch to TensorFlow.

Any suggestions are welcome. I am new to python of this complexity so any pointers would be great. There will be more cleanups and fixes on this PR as I am not finished yet. I need to add some graph plotting and other tidy ups and maybe a refactor. Any guidance would be great. I will also add some docs around this.

If this goes well I might also look at getting pytorch lightning into own part and working in the pipeline.

Please note this will not work with TensorRT or TF lite if there is demand I could look at this in the future.

Cheers

• This is an implementation of a completely new datastore for Donkey. Rather than have a single file for every record, this datastore stores all records in a Seekable file, with O(1) time access. Images are still stored separately.

• This format uses newline delimited json, and stores a configurable number of records per file. It has high level APIs which make it easy to consume records consistently.

• I also ported over tubclean, tubplot and provides a script convert_to_tub_v2.py to move from the old datastore to the new datastore format.

• I ported over the Linear and the Inferred models so far, but adding support for other models should be very trivial.

• I also removed a lot of dead code which should make things a lot easier to understand going forward. For now the old train.py is archived, as we will port over more things from the old training script.

• The new training pipeline also arbitrary pre-processing steps with imgaug which is now a new dependency.

• Added benchmarks for the new datastore.

• Added a new experimental KerasInferred model which learns steering, but not throttle. We then infer throttle based on a inverted exponential curve.

• Added lots of unit tests.

Hi there!

We are running Donkey 4 and found that sometimes our webcam would read in some empty data. They would be very few and far between, say about 4 faulty recordings per 8000 images. We found that when we ran the train script on our data folder, it would crash due to the empty images:

Records # Training 4335 Records # Validation 1084 Epoch 1/100 2021-11-04 20:30:23.748365: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 cannot identify image file 'data/images/492_cam_image_array_.jpg' failed to load image: data/images/492_cam_image_array_.jpg 2021-11-04 20:30:23.946395: W tensorflow/core/framework/op_kernel.cc:1741] Invalid argument: TypeError: %d format: a number is required, not str Traceback (most recent call last):

File "/opt/conda/envs/donkey/lib/python3.7/site-packages/tensorflow/python/ops/script_ops.py", line 243, in call ret = func(*args)

File "/opt/conda/envs/donkey/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py", line 309, in wrapper return func(*args, **kwargs)

File "/opt/conda/envs/donkey/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 785, in generator_py_func values = next(generator_state.get_iterator(iterator_id))

File "/opt/local/donkeycar/donkeycar/pipeline/sequence.py", line 120, in next return self.x_transform(record), self.y_transform(record)

File "/opt/local/donkeycar/donkeycar/pipeline/training.py", line 60, in get_x record, self.image_processor)

File "/opt/local/donkeycar/donkeycar/parts/keras.py", line 241, in x_transform_and_process x_process = img_processor(x_img)

File "/opt/local/donkeycar/donkeycar/pipeline/training.py", line 47, in image_processor img_arr = self.transformation.run(img_arr)

File "/opt/local/donkeycar/donkeycar/pipeline/augmentations.py", line 120, in run aug_img_arr = self.augmentations.augment_image(img_arr)

File "/opt/conda/envs/donkey/lib/python3.7/site-packages/imgaug/augmenters/meta.py", line 766, in augment_image type(image).name),)

TypeError: %d format: a number is required, not str

Traceback (most recent call last): File "train.py", line 31, in main() File "train.py", line 27, in main train(cfg, tubs, model, model_type, comment) File "/opt/local/donkeycar/donkeycar/pipeline/training.py", line 145, in train show_plot=cfg.SHOW_PLOT) File "/opt/local/donkeycar/donkeycar/parts/keras.py", line 183, in train use_multiprocessing=False) File "/opt/conda/envs/donkey/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 66, in _method_wrapper return method(self, *args, **kwargs) File "/opt/conda/envs/donkey/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 848, in fit tmp_logs = train_function(iterator) File "/opt/conda/envs/donkey/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 580, in call result = self._call(*args, **kwds) File "/opt/conda/envs/donkey/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 644, in _call return self._stateless_fn(*args, **kwds) File "/opt/conda/envs/donkey/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 2420, in call return graph_function._filtered_call(args, kwargs) # pylint: disable=protected-access File "/opt/conda/envs/donkey/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1665, in _filtered_call self.captured_inputs) File "/opt/conda/envs/donkey/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1746, in _call_flat ctx, args, cancellation_manager=cancellation_manager)) File "/opt/conda/envs/donkey/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 598, in call ctx=ctx) File "/opt/conda/envs/donkey/lib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute inputs, attrs, num_outputs) tensorflow.python.framework.errors_impl.InvalidArgumentError: TypeError: %d format: a number is required, not str Traceback (most recent call last):

File "/opt/conda/envs/donkey/lib/python3.7/site-packages/tensorflow/python/ops/script_ops.py", line 243, in call ret = func(*args)

File "/opt/conda/envs/donkey/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py", line 309, in wrapper return func(*args, **kwargs)

File "/opt/conda/envs/donkey/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 785, in generator_py_func values = next(generator_state.get_iterator(iterator_id))

File "/opt/local/donkeycar/donkeycar/pipeline/sequence.py", line 120, in next return self.x_transform(record), self.y_transform(record)

File "/opt/local/donkeycar/donkeycar/pipeline/training.py", line 60, in get_x record, self.image_processor)

File "/opt/local/donkeycar/donkeycar/parts/keras.py", line 241, in x_transform_and_process x_process = img_processor(x_img)

File "/opt/local/donkeycar/donkeycar/pipeline/training.py", line 47, in image_processor img_arr = self.transformation.run(img_arr)

File "/opt/local/donkeycar/donkeycar/pipeline/augmentations.py", line 120, in run aug_img_arr = self.augmentations.augment_image(img_arr)

File "/opt/conda/envs/donkey/lib/python3.7/site-packages/imgaug/augmenters/meta.py", line 766, in augment_image type(image).name),)

TypeError: %d format: a number is required, not str

     [[{{node PyFunc}}]]
     [[IteratorGetNext]] [Op:__inference_train_function_1787]

Function call stack: train_function

Our temporary workaround was to run an "ls -l" over the "images" folder and find the faulty recordings, to which we would then delete the problematic catalogues that pointed to those faulty images from the manifest.json file.

I have update my git clone to the nearest version (v 2.5.1), and I have successfully "pip install -e ." and "pip install donkey[pi]", but still got this error. By the way, I tried to mix it by changing "manage.py" file in mycar, instand of "import from controller", I set it to be "from donkeycar.parts.web_controller.web import LocalWebController". This does fix the problem temporarily, but I can't record my data with the recording button pressed on on the website ( I do can control the car on the website). Thanks a lot if anyone have an idea of this issue, this problem hinders three of my groups from processing on. From: Beijing University of Posts and Telecommunications， China

The disk writing on RPi 4 is pretty slow compared to desktops, unless you mount a SSD on the car with it.

The idea is to unblock the TubWriter's IO from main thread. I see a performance boost from average 4ms to 0.2ms: Before (IO on main thread):

+------------------------------+-------+------+------+------+------+-------+-------+
|             part             |  max  | min  | avg  | 50%  | 90%  |  99%  | 99.9% |
+------------------------------+-------+------+------+------+------+-------+-------+
|          TubWriter           | 14.07 | 2.46 | 4.41 | 3.47 | 7.24 | 11.31 | 13.32 |
+------------------------------+-------+------+------+------+------+-------+-------+

After (IO threaded):

+------------------------------+-------+------+------+------+------+------+-------+
|             part             |  max  | min  | avg  | 50%  | 90%  | 99%  | 99.9% |
+------------------------------+-------+------+------+------+------+------+-------+
|          TubWriter           | 79.88 | 0.07 | 0.19 | 0.08 | 0.09 | 0.13 | 21.36 |
+------------------------------+-------+------+------+------+------+------+-------+

Issue description

Steering does not work when using drive , but does in calibration and it works when using an earlier commit. Seeing that another person in discord reported the same problem and that downgrading to 4.2.1 fixed it , I tried a more recent commit d5b91301ff904873e70b9caba72823fe3454feb6. It fixed the problem. I am using a Jetson nano 2gb and the other person has a raspberry pi so it seems to not be related to the particular board used. I went back and reinstalled dev and created a new car to make sure that the problem replicated for me before opening this issue.

Steps to reproduce the issue

1. Follow software setup as described here for the jetson nano
2. create donkeycar from template
3. perform calibration and modify myconfig.py
4. changes to myconfig.py for my car

PCA9685_I2C_BUSNUM = 1
CAMERA_TYPE = "CSIC"   # (PICAM|WEBCAM|CVCAM|CSIC|V4L|D435|MOCK|IMAGE_LIST)
IMAGE_W = 224
IMAGE_H = 224
PCA9685_I2C_BUSNUM = 1
STEERING_LEFT_PWM = 460         #pwm value for full left steering
STEERING_RIGHT_PWM = 260        #pwm value for full right steering
THROTTLE_FORWARD_PWM = 420      #pwm value for max forward throttle
THROTTLE_STOPPED_PWM = 370      #pwm value for no movement
THROTTLE_REVERSE_PWM = 320      #pwm value for max reverse throttle

5. initiate driving from the car folder python manage.py drive
6. navigate to browser
7. press j multiple times to try to turn wheels left
8. press k multiple times to try to turn wheels right
9. also tested on mobile with the touch interface

What's the expected result?

• wheels turn

What's the actual result?

• wheels do not turn

Additional details / screenshot

• the webrowser UI shows changes to the steering, but no changes happen in the car. The throttle still works.
• the working commit d5b91301ff904873e70b9caba72823fe3454feb6 is from December 30th of 2021. I have not tested other commits yet. I figured that someone more familiar with the code would be able to narrow down where the problem originated better than me.
• the jetson nano install instructions is the only one that says to install from dev. The rest say to install from master which seems like it is the same as the most recent release and I am assuming would have also worked for me.
• Should the docs say to install from dev or master for the jetson nano and what should it be for the other devices? @Ezward it looks like you made that change . If you could provide feedback, that would be helpful.

When I followed the docs to convert .h5 to .uff file for TensorRT, I hit an error: AttributeError: module 'tensorflow' has no attribute 'gfile'

It's essentially the same issue from this guy: https://forums.developer.nvidia.com/t/deepstream-object-detector-ssd-cannot-convert-to-uff-file-in-ds5-0ga/145820

After a bit investigation I found the latest TensorRT 7.2.2 only compatible with Tensorflow 1.15: https://docs.nvidia.com/deeplearning/tensorrt/release-notes/tensorrt-7.html#rel_7-2-2

So my workaround is to create another conda environment which has tf 1.15 installed:

conda create -n donkey_tf1 python=3.7
conda activate donkey_tf1
# This will nstall tensorflow 1.15
pip install --pre --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v44 'tensorflow<2'
# Install TensorRT 7.2.2.3, the tarball file method
# https://docs.nvidia.com/deeplearning/tensorrt/install-guide/index.html#installing-tar
# Assuming ${TensorRT-7.2.2.3-Dir} is untar directory
cd ${TensorRT-7.2.2.3-Dir}/python
pip install tensorrt-7.2.2.3-cp37-none-linux_x86_64.whl
cd ${TensorRT-7.2.2.3-Dir}/uff
pip install uff-0.6.9-py2.py3-none-any.whl
cd ${TensorRT-7.2.2.3-Dir}/graphsurgeon
pip install graphsurgeon-0.4.5-py2.py3-none-any.whl
cd ${TensorRT-7.2.2.3-Dir}/onnx_graphsurgeon
pip install onnx_graphsurgeon-0.2.6-py2.py3-none-any.whl

# Now we can use convert-to-uff but has to use the python environment conda provides
~/miniconda3/envs/donkey_tf1/bin/convert-to-uff mypilot.pb

NOTE: if a user want to convert his model to TensorRT, before calling convert-to-uff, the current dev branch also has an issue freezing the model. PR is here: https://github.com/autorope/donkeycar/pull/773

I guess this either need to be addressed or to be documented. TensorRT has great potential and it's sad not very well supported.

The idea is to convert all "prints" into standard Python logging (which by default will look like "prints" in console but can be customized via config) and also sent to MQTT monitoring app.

This is a preliminary PR for review only.

Things to do:

• Convert all "prints" to logger.info(

Adds support for four motors as requested by myself at Freenove/Freenove_4WD_Smart_Car_Kit_for_Raspberry_Pi#41. Strongly based on suggestions by @Ezward. I tested it succesfully on my Freenove 4WD Car.

(my first open source contribution 🎉)

Install Donkeycar on Windows (WSL) The Windows Subsystem for Linux (WSL) lets developers run a GNU/Linux environment -- including most command-line tools, utilities, and applications -- directly on Windows, unmodified, without the overhead of a traditional virtual machine or dualboot setup.

Install Windows Subsystem for Linux. If using Windows 10 (this is not necessary for Windows 11), turn on Windows 10 "Windows Subsystem for Linux" Feature (Settings > Apps > Programs and Features > Turn Windows features on or off) Download a Linux Distribution from the Microsoft Store (recommend Ubuntu Latest) Open the Ubuntu App and configure.

Open the Ubuntu App to get a prompt window via Start Menu | Ubuntu

Install pip using sudo apt install python3-pip

Install the following two libraries for the UI to work: sudo apt-get install libmtdev1 xclip

Change to a directory that you would like to use as the head of all your projects.

mkdir projects cd projects Get the latest donkey from Github. git clone https://github.com/autorope/donkeycar cd donkeycar git checkout main NOTE: The main branch has the latest (unstable) version of donkeycar with experimental features.

Get a stable release from Github: git clone https://github.com/autorope/donkeycar cd donkeycar git fetch --all --tags git checkout tags/4.3.6.1 Install Donkeycar into Python pip3 install -e .[pc] Once you're done with this, close the Ubuntu terminal and then reopen it so the path changes take effect.

Experimental Support - GPU Users: Install Tensorflow GPU - only for NVIDIA Graphics cards If you have an NVIDIA card, you should update to the lastest drivers and install Cuda SDK.

pip3 install tensorflow Create your local working dir: donkey createcar --path /path/to/projects/mycar --template complete

root@DESKTOP-M4INDEG:/home/donkeycar/proj01/donkeycar# donkey createcar --path /mycar donkey: unrecognized option '--path' usage: donkey [-n num] [-f func] sequence seed donkey -i donkey -h | --help | -v | --version | --features

I try to Create a Car in WSL enviroment ,and found that thie donkey command is not support createcar ,and it has it own usage, is that same one ,where is the right one?

Hello, if I have to build a docker image with the current dockerfile project, I get this error:

[C 08:34:18.925 NotebookApp] Bad config encountered during initialization: [C 08:34:18.925 NotebookApp] No such notebook dir: ''/app/notebooks''.

So I propose to solve this error with my current contribution #1073 with commmand

RUN mkdir notebooks into Dockerfile if you don't mind.

Hello, my friend @[email protected] would like to contribut in this project for add docker compose feature. This feature permit using projet easily.