Python package to transfer data in a fast, reliable, and packetized form.

Overview

pySerialTransfer

GitHub version PyPI version

Python package to transfer data in a fast, reliable, and packetized form.

If using this package to communicate with Arduinos, see https://github.com/PowerBroker2/SerialTransfer for the corresponding and compatible library (also available through the Arduino IDE's Libraries Manager).

To Install

pip install pySerialTransfer

Example Python Script

import time
from pySerialTransfer import pySerialTransfer as txfer


if __name__ == '__main__':
    try:
        link = txfer.SerialTransfer('COM17')
        
        link.open()
        time.sleep(2) # allow some time for the Arduino to completely reset
        
        while True:
            send_size = 0
            
            ###################################################################
            # Send a list
            ###################################################################
            list_ = [1, 3]
            list_size = link.tx_obj(list_)
            send_size += list_size
            
            ###################################################################
            # Send a string
            ###################################################################
            str_ = 'hello'
            str_size = link.tx_obj(str_, send_size) - send_size
            send_size += str_size
            
            ###################################################################
            # Send a float
            ###################################################################
            float_ = 5.234
            float_size = link.tx_obj(float_, send_size) - send_size
            send_size += float_size
            
            ###################################################################
            # Transmit all the data to send in a single packet
            ###################################################################
            link.send(send_size)
            
            ###################################################################
            # Wait for a response and report any errors while receiving packets
            ###################################################################
            while not link.available():
                if link.status < 0:
                    if link.status == txfer.CRC_ERROR:
                        print('ERROR: CRC_ERROR')
                    elif link.status == txfer.PAYLOAD_ERROR:
                        print('ERROR: PAYLOAD_ERROR')
                    elif link.status == txfer.STOP_BYTE_ERROR:
                        print('ERROR: STOP_BYTE_ERROR')
                    else:
                        print('ERROR: {}'.format(link.status))
            
            ###################################################################
            # Parse response list
            ###################################################################
            rec_list_  = link.rx_obj(obj_type=type(list_),
                                     obj_byte_size=list_size,
                                     list_format='i')
            
            ###################################################################
            # Parse response string
            ###################################################################
            rec_str_   = link.rx_obj(obj_type=type(str_),
                                     obj_byte_size=str_size,
                                     start_pos=list_size)
            
            ###################################################################
            # Parse response float
            ###################################################################
            rec_float_ = link.rx_obj(obj_type=type(float_),
                                     obj_byte_size=float_size,
                                     start_pos=(list_size + str_size))
            
            ###################################################################
            # Display the received data
            ###################################################################
            print('SENT: {} {} {}'.format(list_, str_, float_))
            print('RCVD: {} {} {}'.format(rec_list_, rec_str_, rec_float_))
            print(' ')
    
    except KeyboardInterrupt:
        try:
            link.close()
        except:
            pass
    
    except:
        import traceback
        traceback.print_exc()
        
        try:
            link.close()
        except:
            pass

Example Arduino Sketch

#include "SerialTransfer.h"


SerialTransfer myTransfer;


void setup()
{
  Serial.begin(115200);
  myTransfer.begin(Serial);
}


void loop()
{
  if(myTransfer.available())
  {
    // send all received data back to Python
    for(uint16_t i=0; i < myTransfer.bytesRead; i++)
      myTransfer.packet.txBuff[i] = myTransfer.packet.rxBuff[i];
    
    myTransfer.sendData(myTransfer.bytesRead);
  }
}

Example Python Script with Callback Functionality

Note that you can specify many callbacks, but only one per packet ID

import time
from pySerialTransfer import pySerialTransfer as txfer


def hi():
    '''
    Callback function that will automatically be called by link.tick() whenever
    a packet with ID of 0 is successfully parsed.
    '''
    
    print("hi")
    
'''
list of callback functions to be called during tick. The index of the function
reference within this list must correspond to the packet ID. For instance, if
you want to call the function hi() when you parse a packet with an ID of 0, you
would write the callback list with "hi" being in the 0th place of the list:
'''
callback_list = [ hi ]


if __name__ == '__main__':
    try:
        link = txfer.SerialTransfer('COM17')
        
        link.set_callbacks(callback_list)
        link.open()
        time.sleep(2) # allow some time for the Arduino to completely reset
        
        while True:
            link.tick()
    
    except KeyboardInterrupt:
        link.close()
    
    except:
        import traceback
        traceback.print_exc()
        
        link.close()
Comments
  • Encoding Many Variables Correctly

    Encoding Many Variables Correctly

    Hello again PowerBroker!

    The lab is happy with how the Serial Transfer works and has asked me to try and implement a transfer of all experimental variables to the Arduino. I'm wondering how I should best transfer many different variables using the package. Should I send things using individual functions one at a time or should I open communications once and send it all during the same transmission? Do you think it matters which way I do it?

    question 
    opened by jmdelahanty 45
  • Saving List onto Arduino

    Saving List onto Arduino

    Hello PowerBroker!

    My name is Jeremy and I just discovered this package for reading/writing to an Arduino using Python. I'm trying to send a list of numbers that indicate a trial type for an experiment I'm working on. Generating the list externally through Python and sending it for use in the experiment would be so helpful for my team!

    I've successfully gotten your package to communicate back and forth to the Arduino, but I'm not sure how to make sure the data I sent is actually saved and usable in the Arduino's memory. I'm not sure if this is the appropriate place to ask about this kind of testing since the package seems to work, but I didn't see a different way of trying to contact you.

    Any advice would be greatly appreciated! It would be cool to contribute to the project as I learn and provide examples relevant for my team's use that others could use.

    Sincerely,

    Jeremy Delahanty Salk Institute for Biological Studies, La Jolla CA

    question 
    opened by jmdelahanty 40
  • Reading is slow

    Reading is slow

    Hey there, I firstly want to say that this library (and its Arduino companion) is awesome and makes communicating with Arduinos so much easier.

    I've been running into some issues reading data from multiple Arduinos at the same time. If I just read from one Arduino, it only takes 1-2ms for transfer.available() to complete (measured using time.perf_counter). But if I connect 2 devices that are both sending data to the computer at 10hz, the readings sometimes only take 1-2ms, but sometimes they take 20-70ms. Usually it starts at 1-2ms for a few seconds, then it starts taking 20-70ms for the rest of the time it is running.

    The 2 transfer objects are on separate threads, so I imagine this is the root of the issue somehow... I am sending data on a separate thread, but I tried adding locks and that didn't fix the problem.

    If I never send any data and just receive, then it always runs at 1-2ms. If I send data for a few seconds and then stop sending data, then it runs at 20-70ms, even well after I stopped sending data.

    Each transfer receives in its own thread in this method:

        def run_serial_receiver(self, remote: transfer.SerialTransfer, lock):
            while True:
                time.sleep(.001)
                with lock:
                    t0 = time.perf_counter()
                    if remote.available():
                        dt = (time.perf_counter() - t0)
                        print(dt)
                        if(dt>.01):
                            print(f"EXCESSIVE TIME: {dt}")
                        sliced_buff = remote.rxBuff[0:remote.bytesRead]
                        rxbytes = bytes(sliced_buff)
                        self.process_RemoteMessage(rxbytes)
    

    And data is sent using this method on the main thread:

        def send_output(self, output):
    
            packet_size = len(output)
            for i, remote in enumerate(self.remotes):
                with self.serial_locks[i]:
                    remote.txBuff = output
                    remote.send(packet_size)
            if self.send_callback:
                self.send_callback()
    

    I'm running on Windows 10, with the SerialTransfer library running on a Teensy 4.0.

    opened by kyleb-imp 22
  • Is there a maximum variable size for

    Is there a maximum variable size for "x" when using myTransfer.sendDatum("x")?

    Hi, I'm trying to send a structure from a microcontroller to my computer. The structure has 7 fields and each field is a "packetSize" element array. Here is the structure initialization on the microcontroller:

    const int packetSize = 5; struct STRUCT { int ax[packetSize]; int ay[packetSize]; int az[packetSize]; int gx[packetSize]; int gy[packetSize]; int gz[packetSize]; int ts[packetSize]; } sendStruct;

    The python code on my computer looks like this: data = link.rx_obj(obj_type=list, obj_byte_size=457, list_format='i')

    In the above python code, 4 is the bytes per number, 5 is the "packetSize," and 7 is the number of fields in my structure. I'm trying to maximize the "packetSize" here. When I upload my arduino code, I can put any number for packetSize, but if I want to receive the data through python the largest packetSize I can use is 9. A packetSize of 9 or less works perfectly, but anything greater than 9 gives the following error:

    // arr = array(list_format, buff) ValueError: bytes length not a multiple of item size //

    Is there a way I can use a packetSize of more than 9 or is this the maximum amount of data that can be sent at once?

    question 
    opened by nolan4 12
  • Receiving duplicate data. Can only read .rx_obj() 7 times before IndexError

    Receiving duplicate data. Can only read .rx_obj() 7 times before IndexError

    Hi, thank you for providing this crucial library! I'm sure I have overlooked something. I appreciate your help in resolving this. Full Arduino code at the bottom of this post.

    The Ultimate Goal: Arduino reads one sample from each of 8 sensors, into an array[8], and transmits the array over serial, 80 times/sec. Python parses the incoming array and plots the values on a graph in real time.

    The good: I believe everything is correct and working well on the arduino side. When I use Serial.print() to print the array of sensor values instead of using SerialTransfer, the values change each time like they should - so the Arduino is not sending duplicate packets.

    The Python Code: (I'm working in Jupyter on Windows10)

    import time
    from pySerialTransfer import pySerialTransfer as txfer
    
    link = txfer.SerialTransfer('COM7', baud=115200)
    link.open()
    time.sleep(2) # allow some time for the Arduino to completely reset
    
    def get_packet(st_obj, byte_size):
        while True:
            st_obj.available()
            yield st_obj.rx_obj(obj_type = list,
                           obj_byte_size = 32,
                           list_format = 'l')
            
    stream = get_packet(link, 32)
    

    I can print 7 times...

    for x in range(7):
        print(stream.__next__())
    

    Prints the same packet 7 times:

    [-179442, -161803, -204057, -239409, -198581, -230921, 16002, -176593] [-179442, -161803, -204057, -239409, -198581, -230921, 16002, -176593] [-179442, -161803, -204057, -239409, -198581, -230921, 16002, -176593] [-179442, -161803, -204057, -239409, -198581, -230921, 16002, -176593] [-179442, -161803, -204057, -239409, -198581, -230921, 16002, -176593] [-179442, -161803, -204057, -239409, -198581, -230921, 16002, -176593] [-179442, -161803, -204057, -239409, -198581, -230921, 16002, -176593]

    If I try to print an 8th time: print(stream.__next__())

    I get an IndexError:

    ---------------------------------------------------------------------------
    IndexError                                Traceback (most recent call last)
    <ipython-input-4-0991908dffd0> in <module>
    ----> 1 stream.__next__()
    
    <ipython-input-2-d12a92ebb86c> in get_packet(st_obj, byte_size)
          5 def get_packet(st_obj, byte_size):
          6     while True:
    ----> 7         st_obj.available()
          8             yield st_obj.rx_obj(obj_type = list,
          9                        obj_byte_size = 32,
    C:\Users\me\AppData\Local\Continuum\anaconda3\lib\site-packages\pySerialTransfer\pySerialTransfer.py in available(self)
        475                     elif self.state == find_payload:
        476                         if self.payIndex < self.bytesToRec:
    --> 477                             self.rxBuff[self.payIndex] = recChar
        478                             self.payIndex += 1
        479 
    
    IndexError: list assignment index out of range
    

    What am I doing wrong here?

    Full Arduino Code:

    #include "HX711-multi.h"
    #include "SerialTransfer.h"
    
    // Hardware Config
    #define CLK1 2    // clock pin to the first group of ADCs
    #define DIN1 3    // data pin to the first lca
    #define DIN2 4    // data pin to the second lca
    #define DIN3 5    // data pin to the third lca
    #define DIN4 6    // data pin to the fourth lca
    
    #define CLK2 8    // clock pin to the 2nd group of ADCs
    #define DIN5 7    // data pin to the fifth lca
    #define DIN6 10   // data pin to the sixth lca
    #define DIN7 11   // data pin to the seventh lca
    #define DIN8 12   // data pin to the eighth lca
    
    // System Config
    #define CHANNEL_COUNT 4
    #define NUM_GROUPS 2
    #define GAIN 128                  // Gain level for HX711 to use ( 32, 64, 128 are your options)
    
    // Construct scale objects
    byte DINS1[CHANNEL_COUNT] = {DIN1, DIN2, DIN3, DIN4};
    HX711MULTI scales1(CHANNEL_COUNT, DINS1, CLK1, GAIN);
    
    byte DINS2[CHANNEL_COUNT] = {DIN5, DIN6, DIN7, DIN8};
    HX711MULTI scales2(CHANNEL_COUNT, DINS2, CLK2, GAIN);
    
    HX711MULTI scales[NUM_GROUPS] = {scales1, scales2};      // Add all scale instances to an array so we can iterate over them
    
    // Program Variables
    long raw[NUM_GROUPS * CHANNEL_COUNT];      // Array to hold raw samples collected from each sensor
    uint16_t sendSize;
    int i;
    
    SerialTransfer myTransfer;
    
    void setup() {
      Serial.begin(115200);
      myTransfer.begin(Serial);
    }
    
    void loop() {
      //Read sensor values into an array
      scales[0].readRaw(raw);                  // Read values from first 4 sensors directly into the raw array
      scales[1].readRaw(raw + 4);           // Read values from last 4 sensors directly into the raw array
      // contents of raw looks like: {-180279, -160207, 57267, -238712, -198426, -232254, 14375, -176826}
    
      // Transfer the contents of raw to Python via serial
      sendSize = myTransfer.txObj(raw, sendSize);
      myTransfer.sendData(sendSize);
    
      // Loop around, read new values from sensors, and send new values.
    }
    
    question 
    opened by dunderMethods 10
  • Problem with the example scripts

    Problem with the example scripts

    Hello!

    The Example Python Script works well with Arduino Nano 33 BLE Sense.

    However, when I try to modify it to send an array of floats, using the "Send a list" way, the tx_obj() -method returns the following error:

    Traceback (most recent call last):
      File "C:\Users\Klupi\AppData\Local\Temp\ipykernel_25544\2144615584.py", line 19, in <module>
        list_size = link.tx_obj(list_)
      File "C:\Users\Klupi\miniconda3\envs\deeplearning\lib\site-packages\pySerialTransfer\pySerialTransfer.py", line 272, in tx_obj
        start_pos = self.tx_obj(el, start_pos)
      File "C:\Users\Klupi\miniconda3\envs\deeplearning\lib\site-packages\pySerialTransfer\pySerialTransfer.py", line 289, in tx_obj
        self.txBuff[index + start_pos] = val_bytes[index]
    IndexError: list assignment index out of range
    

    Why is this, what am I doing wrong? My sample array is as follows:

    [-0.27273, -0.33333, 0.066667, -0.10638, -0.27273, -0.33333, -0.066667, -0.10638, -0.045455, -0.28889, -0.022222, 0.021277, -0.045455, -0.46667, -0.2, 0.10638, -0.045455, -0.2, 0.066667, 0.14894, -0.045455, -0.33333, 0.42222, 0.14894, -0.59091, -0.33333, 0.33333, 0.021277, -0.59091, -0.28889, 0.066667, -0.31915, -0.59091, -0.28889, -0.55556, 0.021277, -0.59091, -0.24444, -0.066667, 0.021277, -0.18182, 0.066667, 0.11111, 0.14894, -0.18182, 0.066667, 0.066667, 0.14894, -0.18182, 0.022222, -0.022222, -0.021277, -0.18182, 0.066667, 0.066667, -0.021277, 0.13636, 0.066667, -0.022222, -0.021277, 0.36364, -0.33333, 0.022222, -0.021277, 0.090909, -0.066667, -0.33333, -0.44681, 0.090909, -0.42222, -0.2, -0.19149, 0.090909, -0.11111, -0.11111, 0.10638, 0.090909, -0.24444, 0.022222, -0.021277, 0.090909, -0.066667, -0.066667, -0.021277, 0.090909, 0.066667, 0.066667, -0.021277, 0.090909, -0.022222, -0.2, 0.021277, 0.090909, -0.46667, -0.33333, 0.021277, 0.090909, -0.066667, -0.33333, -0.3617, 0.090909, -0.066667, -0.2, -0.3617, 0.090909, -0.066667, -0.33333, -0.48936, 0.090909, -0.24444, -0.2, -0.48936, -0.18182, -0.68889, -0.55556, -0.3617, -0.18182, -0.28889, -0.33333, -0.3617, -0.090909, -0.022222, -0.42222, -0.3617, 0.0, -0.24444, -0.86667, -0.3617, -0.045455, -0.2, -0.2, -0.10638, -0.045455, -0.066667, -0.33333, -0.10638, 0.090909, -0.11111, -0.33333, -0.65957, 0.090909, 0.2, -0.73333, -0.3617, 0.090909, 0.33333, -0.73333, -0.3617, 0.13636, 0.066667, -0.91111, -0.3617, 0.22727, -0.15556, -0.68889, -0.44681, 0.22727, 0.24444, -0.55556, -0.44681, 0.090909, 0.28889, -0.6, -0.44681, 0.63636, 0.11111, -0.42222, -0.74468, 0.63636, 0.066667, -0.6, -0.74468, 0.72727, 0.33333, -0.37778, -0.61702, 0.72727, -0.066667, -0.6, -0.61702, 0.81818, 0.2, -0.77778, -0.61702, 0.81818, 0.2, -0.73333, -0.61702, 0.77273, 0.066667, -0.95556, -0.61702, 0.77273, -0.2, -0.82222, -0.61702, 0.81818, 0.022222, -0.95556, -0.61702, 0.77273, 0.066667, -0.6, -0.82979, 0.77273, 0.33333, -0.73333, -0.82979, 0.77273, -0.37778, -0.86667, -0.82979, 0.27273, -0.066667, -0.68889, -0.3617, 0.27273, -0.066667, -0.68889, -0.70213, 0.27273, 0.15556, -0.6, -0.70213, 0.27273, 0.33333, -1.0, -0.70213, 0.77273, -0.15556, -0.82222, -0.61702, 0.77273, 0.6, -1.0, -0.61702, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
    

    Interestingly, sending a smaller array such as [1.2433333333334, 2.34222222221, 3.4255555555555] works fine. How would you proceed on sending the bigger array to Arduino?

    Many thanks!

    question 
    opened by eppane 7
  • CRC ERROR keep occuring

    CRC ERROR keep occuring

    Hello powrebroker

    First of all really big thanks to your awesome library.

    As I'm using your library for transmitting simulator data (python) to arduino(mega) CRC Error keep occur and pySerialTransfer Thread keep waiting for timeout and send CRC Error. I lower the baud rate to 9600 and provide some delay to work properly

    here Is my code 6

    5

    2

    question 
    opened by jaykorea 7
  • Send encoder data

    Send encoder data

    Hello, I am building mobile robot based on nvidia jetson nano and arduino. I have two motor encoders connected to arduino so I need to send data to jetson nano (python) via uart. What is the best and right way to do it with this library? Approximate range of encoder values is -1000000 - 1000000. I will appreciate your help.

    question 
    opened by w1ngedshadow 6
  • Parsing char from Arduino side

    Parsing char from Arduino side

    Hello PB2 and a big thank you for your work!

    I'm trying to trigger some action in the Arduino Leonardo side when a specific char is received from python. I could have use int instead of char message but I take this as an exercice :)

    All the manipulation I tried failed... I'm starting to be tired of this and probably missing some perspective from this point.

    So, here my Python:

    X1 = [0, 0, 0, 0, 0, ... , 0]
    X2 = [1, 1, 1, 1, 1, .... , 1] ---> X1 and X2 are long array
    
    try:
        link = txfer.SerialTransfer('COM8', 115200, debug=True)
        link.open()
    
        start = 0
        step = 10
        counter = 0
    
        while counter < len(X1):
            sendSize = 0
    
            X1_= X1[start:start+step] ---> I split it in order to get under 254 bytes per packet.
            X2_= X2[start:start+step]
            message = "notes"
            
            messageSize = link.tx_obj(message) 
            sendSize += messageSize
    
            X1Size = link.tx_obj(X1_, sendSize) - sendSize
            sendSize += X1Size
    
            X2Size = link.tx_obj(X2, sendSize) - sendSize
            sendSize += X2Size
    
            link.send(sendSize, packet_id=0) ---> I recently tried to add packet_id information to don't messing up the receiving part.
            counter += len(X1_)
    
            #print('SENT: {} {} {}'.format(message, X1_, X2_))
    
            while not link.available():
                pass
    
            rec_message = link.rx_obj(obj_type=type(message), obj_byte_size=messageSize)
            rec_X1 = link.rx_obj(obj_type=type(X1_), obj_byte_size=X1Size, list_format='i', start_pos = messageSize)
            rec_X2 = link.rx_obj(obj_type=type(X2_), obj_byte_size=X2Size, list_format='i', start_pos = messageSize + X1Size)
    
            start += step
            print("Recu:\t{} - {} et {}".format(rec_message, rec_X1, rec_X2))
    
    
        while True:
            sendSize2 = 0
            message2 = "click"
            messageSize = link.tx_obj(message2)
            sendSize2 += messageSize
    
            print("Ready to click...")
            link.send(sendSize2, packet_id=0)
            #print('SENT: {}'.format(message2))
    
            while not link.available():
                pass
    
            rec_message = link.rx_obj(obj_type=type(message2), obj_byte_size=messageSize)
            print("Recu:\t{}".format(rec_message))
    
            raise KeyboardInterrupt
    
    except KeyboardInterrupt:
        try:
            link.close()
        except:
            pass
    
    except:
        import traceback
        traceback.print_exc()
    
        try:
            link.close()
        except:
            pass
    

    And my Arduino:

    #include "SerialTransfer.h"
    #include "Keyboard.h"
    #include "Mouse.h"
    
    SerialTransfer myTransfer;
    
    int32_t X1[10];
    int32_t X2[10];
    
    int32_t X1G[] = {};
    int32_t X2G[] = {};
    
    char message[5];
    
    #Stuff I tried....
    char notes[] = "notes";
    char clickOk[] = "click";
    
    void setup()
    {
      Keyboard.begin();
      Mouse.begin();
      Serial.begin(115200);
      myTransfer.begin(Serial, true);
    }
    
    void loop()
    {
      if(myTransfer.available())
      {
        uint16_t recSize = 0;
        recSize = myTransfer.rxObj(message, recSize);
    
        if (strcmp(message, clickOk) == 0) ---> Here my problem. How to check the received message and do action?
        {
          recSize = myTransfer.rxObj(X1, recSize);
          recSize = myTransfer.rxObj(X2, recSize);
      
          memmove(X1G + sizeof(X1G), noteTime, sizeof(X1));
          memmove(X2G + sizeof(X2G), noteType, sizeof(X2));
      
          uint16_t sendSize = 0;
          
          sendSize = myTransfer.txObj(message, sendSize);
          sendSize = myTransfer.txObj(X1, sendSize);
          sendSize = myTransfer.txObj(X2, sendSize);
          
          myTransfer.sendData(sendSize);
         
          delay(50);
        }
        else if (strcmp(message, clickOk) == 0)
        {
          uint16_t sendSize2 = 0;
          sendSize2 = myTransfer.txObj(message, sendSize2);
          myTransfer.sendDatum(sendSize2);
          Mouse.click();
        }
      }
    }
    

    My FTDI card will be coming soon, it should be easier for debugging, but in the meantime any help will be appreciated :)

    I'm a beginner in this wide world. Maybe my mechanism is completely broken, so feel free to suggest another solution to get what I want.

    Bye, Vouncat

    question 
    opened by Vounecat 6
  • pySerialTransfer Questions

    pySerialTransfer Questions

    Hello PowerBroker2,

    We are a high school robotics team that came across your code. One of our goals is to run serial communication between Python on a laptop and an Arduino over RS485. We have tried your pySerialTransfer code to verify functionality.

    We are able to get Python to send and receive data while using the USB connection to the Arduino, but not over RS485. At first, we thought our RS485 communication could be a hardware issue. However, all of our hardware appears to work individually.

    1. We have tested our USB to RS485 adapter using a second adapter and two instances of "AccessPort" from Serialcomm.com. We are able to communicate between two laptops with this method.

    2. We replaced the USB to RS485 adapters using two Arduinos connected through their serial pins. The Arduinos both run an "empty" sketch with pins 0 and 1 set as inputs. We are able to communicate between two laptops with AccessPort as above.

    3. We have tested two Arduinos communicating over RS485 using TTL to RS485 converters. These work as well.

    4. We have tried using a USB to RS485 adapter with A, B and ground running to the TTL to RS485 converter. We are using Serial1 for RS485 communication and Serial to send debug statements to the Serial Monitor. The code never enters the "if(myTransfer.available())" loop on the Arduino.

    5. To try to simplify the connections, our latest test was to eliminate the RS485 connections. We returned to using two Arduinos. One Arduino used the setup described in "2" above to serve as a USB to TTL converter. The second Arduino ran your sample code. The result is the same as in "4" above. In both of these cases, we modified the code to use Serial1. After the "if(myTransfer.available())" we added "else Serial.println("Not available");" to give us feedback.

    After reading through several web pages that you have offered advice on, we came across issue #32 (https://github.com/PowerBroker2/pySerialTransfer/issues/32). We tried several of the code examples you provide there with no luck.

    Software is our weakest area this year. The addition of Python has made it more complicated for us. We apologize if there is a simple solution that we are missing.

    We are not understanding much of the code as it relates to SerialTransfer and pySerialTransfer, so we do not know how to further analyze the "if(myTransfer.available())" functionality. We have started reading through "Serial Input Basics" and your "Serial Input Advanced" tutorials on the Arduino website.

    Another issue that came up for us was in trying one of your examples shared here: (https://forum.arduino.cc/t/arduino-to-arduino-serial/621141/15). Our Arduino would not compile your code and gave us the following error: "class SerialTransfer' has no member named 'txBuff'"

    With all that said, we are wondering if you have any suggestions since pySerialTransfer seems like it would be helpful to us.

    Thanks in advance for your time.

    help wanted 
    opened by hawkseng 6
  • Float problem

    Float problem

    Hi im got a problem with float on python side (RX), got a this kind of output: b'P'2.1579996350602183e-43 When im sending from uC: testStruct.z = 'P'; testStruct.y = 4.55; Betwen uC is no problem work like a harm, on python floats crazy. 9600 baud / Python 3.9.2 / Debian

    question 
    opened by MarcinKlima 6
  • Check for CRC error

    Check for CRC error

    Hello,

    thank you for this great package!

    I wanted to understand your code and manipulated some lines in CRC.py.

    But with your example Python script, nothing printed and I didn't get a Python error. After a while, I realized that:

    CRC_ERROR = 0

    So I think in the example script it should be:

    if link.status <= 0:

     while not link.available():
                    if link.status <= 0:
                        if link.status == txfer.CRC_ERROR:
                            print('ERROR: CRC_ERROR')
                        elif link.status == txfer.PAYLOAD_ERROR:
                            print('ERROR: PAYLOAD_ERROR')
                        elif link.status == txfer.STOP_BYTE_ERROR:
                            print('ERROR: STOP_BYTE_ERROR')
                        else:
                            print('ERROR: {}'.format(link.status))
    

    Kind regards, 8sawte8

    bug 
    opened by 8sawte8 1
  • Want support for transferring bytes()

    Want support for transferring bytes()

    bytes() is its own type in python3, and is common enough IMHO that pySerialTransfer should support it. Since bytes() do not need packing with struct.pack(), the proposed solution is to change the logic in txObj to check for type(val) == bytes and not attempt to pack it.

    Currently, byte strings must be unpacked into ints then passed in with one call to txObj per byte, giving val_type_override as: send_size = link.tx_obj(self.left_drive_pct, val_type_override='b')

    The alternative - to send values as ints and convert them to bytes on the Arduino side, would consume 4X the bandwidth on the serial link.

    Thoughts? I am willing to contribute this fix in a pull request if you support the idea.

    opened by PaulBouchier 1
  • Arduino tx_data + pySerialTransfer rx_data example fails with esp32 Arduino

    Arduino tx_data + pySerialTransfer rx_data example fails with esp32 Arduino

    On multi-byte wide processors the compiler aligns data items to their native alignment, unless directed otherwise. Thus a float is aligned to a 4-byte boundary. Therefore the testStruct in tx_data is laid out in memory differently between an Arduino Uno & a board like an esp32. Since txObj uses sizeof(T) as the number of bytes to transmit, tx_data sends a different number of bytes from esp32 compared to Uno.

    testStruct is shown below with the fix needed for processors with > 8-bit wide data bus struct STRUCT { char z; float y; //} testStruct; } attribute((packed)) testStruct;

    The original code (commented out line 4 above) causes 3 empty bytes to be placed after z, and before y on esp32, so txObj sends 8 bytes on an esp32, but only 5 on an Arduino Uno. pySerialTransfer is unaware of the packing difference, so tries to pull the float out of bytes 1-4 (counting byte 0 as z), regardless of the sender-architecture.

    The example as provided prints the following correct message on pySerialTransfer + Uno: b'$'4.5 | hello but the following error message on pySerialTransfer with esp32, caused by pySerialTransfer reading the float out of bytes 1-4 then trying to read the char array out of byte 5, which is actually the 2nd byte of the float.: Traceback (most recent call last): File "./rx_data.py", line 30, in arr = link.rx_obj(obj_type=str, File "/home/bouchier/.local/lib/python3.8/site-packages/pySerialTransfer/pySerialTransfer.py", line 359, in rx_obj unpacked_response = unpacked_response.decode('utf-8') UnicodeDecodeError: 'utf-8' codec can't decode byte 0x90 in position 1: invalid start byte

    Removing the string transfer part of the example prints the following incorrect message on pySerialTransfer + esp32, : b'$'0.0 | Note the float, which should be 4.5, is 0.0 because pySerialTransfer is reading the float out of the 3 empty bytes plus first byte of the float.

    The correct way to eliminate machine/compiler dependent packing is with the packed attribute as shown on line 5 of testStruct above. Making this change produces the correct output on both Uno & esp32: b'$'4.5 | hello

    Are you open to pull requests to fix the examples so they work on both Uno & esp32? I would test any such changes on both.

    opened by PaulBouchier 6
  • Examples use double on Arduino side, format 'f' on python side

    Examples use double on Arduino side, format 'f' on python side

    Hey PowerBroker - nice library! Thanks very much - it will save me much work compared with PacketSerial.

    The last checkin comment for examples/datum/Arduino/ was that it fixed float precision. But Arduino declares doubles in the datum and data examples, but the python side reads it as 'f' (float). Double size is apparently platform-dependent; ghe Arduino language reference says double is 4 bytes (same as float) on Uno and ATMEGA but 8 bytes on Due. I am testing it on an esp32, where apparently doubles are 8 bytes, because I had to change the format specifier in the python example code to 'd' to get it to read the data from esp32-Arduino correctly.

    I would think you should make the types match on the Arduino & python sides. Is there some deeper issue here that made you declare Due unsupported? Would such an issue also apply to esp32?

    And on the subject of esp32, does the design take care to keep tx & rx logic in Arduino & python (and everything they use) separate such that a receive thread and a transmit thread could run through the classes simultaneously and not step on each other?

    question 
    opened by PaulBouchier 9
  • Multiple Packets for One Array

    Multiple Packets for One Array

    Hey PowerBroker!

    Things are moving along pretty smoothly with our project! I can send trial sets of 45 or smaller no problem and the lab is pretty happy with it. A new challenge has appeared for me, though. The lab would like it to be possible to double the amount of trials to 90 total as a definite ceiling.

    I had thought that I was doing this correctly with what you have taught me so far but, after some testing, I've come to realize that while a second packet is getting sent by Python, my Arduino code doesn't seem to collecting the second packet's content. Here's an example of what I mean:

    (bruker_control) C:\Users\jdelahanty>python Documents\gitrepos\headfix_control\bruker_control\bruker_control.py
    
    First Half of Trials Sent
    [1, 1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0]
    First Half of Trials Received
    [1, 1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0]
    First Half Trial Array Transfer Successful!
    Second Half of Trials Sent
    [0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1]
    Second Half of Trials Received
    [1, 1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0] # <--- Arduino sends first packet's contents!
    Second Half Trial Array Transfer Failure!
    Exiting...
    Traceback (most recent call last):
      File "Documents\gitrepos\headfix_control\bruker_control\bruker_control.py", line 522, in serial_transfer_trialArray
        sys.exit()
    SystemExit
    Experiment Over!
    Exiting...
    

    Any advice? I had thought it would collect the packet independently but I'm clearly misunderstanding how multiple packets for the same object work. Is this something I can do even or should I create a second array in the Arduino code to store the second half of trials?

    Here's some full Python and Arduino code: Python

    #### Trial Array Generation ####
    # Import scipy for statistical distributions
    import scipy
    # Import scipy.stats truncated normal distribution for ITI Array
    from scipy.stats import truncnorm
    # Import numpy for trial array generation/manipulation and Harvesters
    import numpy as np
    # Import numpy default_rng
    from numpy.random import default_rng
    
    # -----------------------------------------------------------------------------
    # Functions
    # -----------------------------------------------------------------------------
    #### Trial Generation ####
    # Random Trials Array Generation
    def gen_trial_array(totalNumberOfTrials):
        # Always initialize trial array with 3 reward trials
        trialArray = [1,1,1]
        # Define number of samples needed from generator
        num_samples = totalNumberOfTrials - len(trialArray)
        # Define probability that the animal will receive sucrose 60% of the time
        sucrose_prob = 0.5
        # Initialize random number generator with default_rng
        rng = np.random.default_rng(2)
        # Generate a random trial array with Generator.binomial
        # Use n=1 to pull one sample at a time, p=.6 as probability of sucrose
        # Use num_samples to fill out accurate number of trials
        # Use .tolist() to convert random_trials from np.array to list
        random_trials = rng.binomial(
        n=1, p=sucrose_prob, size=num_samples
        ).tolist()
        # Append the two arrays together
        for i in random_trials:
            trialArray.append(i)
    
        if len(trialArray) > 45:
            split_array = np.array_split(trialArray, 2)
            first_trialArray = split_array[0].tolist()
            second_trialArray = split_array[1].tolist()
        else:
            first_trialArray = None
            second_trialArray = None
    
        ## TODO: Write out the trial array into JSON as part of experiment config
        # Return trialArray or halves of trial arrays (if needed)
        return trialArray, first_trialArray, second_trialArray
    
    #### Serial Transfer ####
    # Import pySerialTransfer for serial comms with Arduino
    from pySerialTransfer import pySerialTransfer as txfer
    
    
    # Trial Array Transfer
    def serial_transfer_trialArray(trialArray, first_trialArray, second_trialArray):
        # Check if two packets are necessary. If 'first_trialArray' is None,
        # the message is small enough to fit in one packet. There's no need to
        # check the second_trialArray as it too will be None by design in this case.
        if first_trialArray == None:
            try:
                # Initialize COM Port for Serial Transfer
                link = txfer.SerialTransfer('COM12', 115200, debug=True)
    
                # Send the trial array
                # Initialize trialArray_size of 0
                trialArray_size = 0
                # Stuff packet with size of trialArray
                trialArray_size = link.tx_obj(trialArray)
                # Open communication link
                link.open()
                # Send array
                link.send(trialArray_size, packet_id=0)
    
                print(trialArray)
    
                while not link.available():
                    pass
    
                # Receive trial array:
                rxtrialArray = link.rx_obj(obj_type=type(trialArray),
                obj_byte_size=trialArray_size, list_format='i')
    
                print(rxtrialArray)
    
                if trialArray == rxtrialArray:
                    print("Trial Array transfer successful!")
    
                else:
                    link.close()
                    print("Trial Array error! Exiting...")
                    sys.exit()
    
                # Close the communication link
                link.close()
    
            except KeyboardInterrupt:
                try:
                    link.close()
                except:
                    pass
    
            except:
                import traceback
                traceback.print_exc()
    
                try:
                    link.close()
                except:
                    pass
    
        elif first_trialArray != None:
            try:
                # Initialize COM Port for Serial Transfer
                link = txfer.SerialTransfer('COM12', 115200, debug=True)
    
                # Send the first half of trials with packet_id = 0
                first_trialArray_size = 0
                first_trialArray_size = link.tx_obj(first_trialArray)
                link.open()
                link.send(first_trialArray_size, packet_id=0)
    
                print("First Half of Trials Sent")
                print(first_trialArray)
    
                while not link.available():
                    pass
    
                # Receive the first half of trials from Arduino
                rxfirst_trialArray = link.rx_obj(obj_type=type(first_trialArray),
                obj_byte_size=first_trialArray_size, list_format='i')
    
                print("First Half of Trials Received")
                print(rxfirst_trialArray)
    
                # Confirm packet was sent correctly
                if first_trialArray == rxfirst_trialArray:
                    print("First Half Trial Array Transfer Successful!")
                else:
                    link.close()
                    print("First Half Trial Array Transfer Failure!")
                    print("Exiting...")
                    sys.exit()
    
                # Send second half of trials with packet_id = 0
                second_trialArray_size = 0
                second_trialArray_size = link.tx_obj(second_trialArray)
                link.send(second_trialArray_size, packet_id=1)
    
                print("Second Half of Trials Sent")
                print(second_trialArray)
    
                # Receive second half of trials from Arduino
                rxsecond_trialArray = link.rx_obj(obj_type=type(second_trialArray),
                obj_byte_size=second_trialArray_size, list_format='i')
    
                print("Second Half of Trials Received")
                print(rxsecond_trialArray)
    
                if second_trialArray == rxsecond_trialArray:
                    print("Second Half Trial Array Transfer Successful!")
                else:
                    link.close()
                    print("Second Half Trial Array Transfer Failure!")
                    print("Exiting...")
                    sys.exit()
    
                link.close()
    
            except KeyboardInterrupt:
                try:
                    link.close()
                except:
                    pass
    
            except:
                import traceback
                traceback.print_exc()
    
                try:
                    link.close()
                except:
                    pass
        else:
            print("Something is wrong...")
            print("Exiting...")
            sys.exit()
    

    Arduino

    // Use package SerialTransfer.h from PowerBroker2 https://github.com/PowerBroker2/SerialTransfer
    #include "SerialTransfer.h"
    
    // Rename SerialTransfer to myTransfer
    SerialTransfer myTransfer;
    
    const int MAX_NUM_TRIALS = 46; // maximum number of trials possible
    
    int32_t trialArray[MAX_NUM_TRIALS]; // create trial array
    
    boolean acquireTrials = true;
    
    void setup()
    {
      Serial.begin(115200);
      Serial1.begin(115200);
    
      myTransfer.begin(Serial1, true);
    }
    
    
    void loop(){
      trials_rx();
    }
    
    int trials_rx() {
      if (acquireTrials) {
        if (myTransfer.available())
        {
          myTransfer.rxObj(trialArray);
          Serial.println("Received Trial Array");
    
          myTransfer.sendDatum(trialArray);
          Serial.println("Sent Trial Array");
    
          acquireTrials = false;
        }
      }
    }
    
    question 
    opened by jmdelahanty 29
Releases(2.6.7)
Owner
PB2
Electrical Engineer (Virginia Tech alumnus)
PB2
VevestaX is an open source Python package for ML Engineers and Data Scientists.

VevestaX Track failed and successful experiments as well as features. VevestaX is an open source Python package for ML Engineers and Data Scientists.

Vevesta 24 Dec 14, 2022
GWpy is a collaboration-driven Python package providing tools for studying data from ground-based gravitational-wave detectors

GWpy is a collaboration-driven Python package providing tools for studying data from ground-based gravitational-wave detectors. GWpy provides a user-f

GWpy 342 Jan 7, 2023
Python package for processing UC module spectral data.

UC Module Python Package How To Install clone repo. cd UC-module pip install . How to Use uc.module.UC(measurment=str, dark=str, reference=str, heade

Nicolai Haaber Junge 1 Oct 20, 2021
PyEmits, a python package for easy manipulation in time-series data.

PyEmits, a python package for easy manipulation in time-series data. Time-series data is very common in real life. Engineering FSI industry (Financial

Thompson 5 Sep 23, 2022
nrgpy is the Python package for processing NRG Data Files

nrgpy nrgpy is the Python package for processing NRG Data Files Website and source: https://github.com/nrgpy/nrgpy Documentation: https://nrgpy.github

NRG Tech Services 23 Dec 8, 2022
Python package for analyzing behavioral data for Brain Observatory: Visual Behavior

Allen Institute Visual Behavior Analysis package This repository contains code for analyzing behavioral data from the Allen Brain Observatory: Visual

Allen Institute 16 Nov 4, 2022
PLStream: A Framework for Fast Polarity Labelling of Massive Data Streams

PLStream: A Framework for Fast Polarity Labelling of Massive Data Streams Motivation When dataset freshness is critical, the annotating of high speed

null 4 Aug 2, 2022
Amundsen is a metadata driven application for improving the productivity of data analysts, data scientists and engineers when interacting with data.

Amundsen is a metadata driven application for improving the productivity of data analysts, data scientists and engineers when interacting with data.

Amundsen 3.7k Jan 3, 2023
Elementary is an open-source data reliability framework for modern data teams. The first module of the framework is data lineage.

Data lineage made simple, reliable, and automated. Effortlessly track the flow of data, understand dependencies and analyze impact. Features Visualiza

null 898 Jan 9, 2023
A powerful data analysis package based on mathematical step functions. Strongly aligned with pandas.

The leading use-case for the staircase package is for the creation and analysis of step functions. Pretty exciting huh. But don't hit the close button

null 48 Dec 21, 2022
small package with utility functions for analyzing (fly) calcium imaging data

fly2p Tools for analyzing two-photon (2p) imaging data collected with Vidrio Scanimage software and micromanger. Loading scanimage data relies on scan

Hannah Haberkern 3 Dec 14, 2022
🧪 Panel-Chemistry - exploratory data analysis and build powerful data and viz tools within the domain of Chemistry using Python and HoloViz Panel.

???? ??. The purpose of the panel-chemistry project is to make it really easy for you to do DATA ANALYSIS and build powerful DATA AND VIZ APPLICATIONS within the domain of Chemistry using using Python and HoloViz Panel.

Marc Skov Madsen 97 Dec 8, 2022
Fast, flexible and easy to use probabilistic modelling in Python.

Please consider citing the JMLR-MLOSS Manuscript if you've used pomegranate in your academic work! pomegranate is a package for building probabilistic

Jacob Schreiber 3k Jan 2, 2023
Sensitivity Analysis Library in Python (Numpy). Contains Sobol, Morris, Fractional Factorial and FAST methods.

Sensitivity Analysis Library (SALib) Python implementations of commonly used sensitivity analysis methods. Useful in systems modeling to calculate the

SALib 663 Jan 5, 2023
A python package which can be pip installed to perform statistics and visualize binomial and gaussian distributions of the dataset

GBiStat package A python package to assist programmers with data analysis. This package could be used to plot : Binomial Distribution of the dataset p

Rishikesh S 4 Oct 17, 2022
ToeholdTools is a Python package and desktop app designed to facilitate analyzing and designing toehold switches, created as part of the 2021 iGEM competition.

ToeholdTools Category Status Repository Package Build Quality A library for the analysis of toehold switch riboregulators created by the iGEM team Cit

null 0 Dec 1, 2021
Created covid data pipeline using PySpark and MySQL that collected data stream from API and do some processing and store it into MYSQL database.

Created covid data pipeline using PySpark and MySQL that collected data stream from API and do some processing and store it into MYSQL database.

null 2 Nov 20, 2021
Python data processing, analysis, visualization, and data operations

Python This is a Python data processing, analysis, visualization and data operations of the source code warehouse, book ISBN: 9787115527592 Descriptio

FangWei 1 Jan 16, 2022
A Python package for Bayesian forecasting with object-oriented design and probabilistic models under the hood.

Disclaimer This project is stable and being incubated for long-term support. It may contain new experimental code, for which APIs are subject to chang

Uber Open Source 1.6k Dec 29, 2022