Call-graph profiling for TwinCAT 3

Overview

Twingrind

This project brings profiling to TwinCAT PLCs. The general idea of the implementation is as follows.

  1. Twingrind is a TwinCAT library that includes a program, which is used for profiling. It includes methods to built-up a callstack and some triggers to start profiling.
  2. prepare.py is a python script that is used to add profiling boilerplate code to your PLC.
  3. fetch.py can be used to read callstack from the PLC
  4. reconstruct.py is used to convert the recorded callstacks to callgrind (http://kcachegrind.sourceforge.net/html/CallgrindFormat.html)

The profile can then be visualized by qcachegrind. What follows is a short instruction how to use the code that is provided in this repository for profiling your PLC. The following image shows a visualization of the callstack of a single PLC cycle - Methodnames have been obfuscated

Callgrind demo

The profiler can be set up to only capture callstacks of PLC cycles, which take "too long", which is adjustable by a threshold that is relative to the PLC cycletime. This feature is super handy for finding realtime violation issues where the cycletime is exceeded.

The implementation is by no means feature complete and far from perfect. The core of the implementation was written way before TwinCAT offered any kind of profile mechanism, and I actually had the needs of a profile to find a serious problem in a PLC. Nowadays profiling for TwinCAT is offered by Beckhoff, but is of course attached with licencing fees and subpar visualization (my two cents). Twingrind instead uses a common fileformat for profiling and is free software. If you are interested to contribute to the project, feel free to write issues or fork the project and create pull requests. The current limitations of the profiler and the topics, which should be looked in, are as follows.

  • Only methods are profiled, calls to function blocks are not implemented yet. > Support for methods and functionblock calls and functions
  • Only 1 task and 1 PRG are supported.
  • Profiling itself adds some overhead to your code, which can not be avoided by this method, but maybe reduced by a fair bit.
  • Calls in referenced PLC methods are not profiled and at the moment there is no way to add the profiling boilerplate code to PLC libraries, which are used by your PLC > prepare.py can now be called for library plcs and the hashmap file can be reused
  • No unicode character support in source files reading files automatically detects its encoding and write the source code back with the same encoding

Backup

Before profiling you should backup your code by commiting it to your version-control system or at least copy&paste it to a different location. The script will modify your existing source code and although it has been tested thoroughly, it is always better to err on the side of caution.

Install

Twingrind can either be downloaded from Github as or you can clone the repository and compile the library yourself. This guide will focus on the former use case.

First, get the latest release of the Twingrind, the download will give you a file called "twingrind_0.1.1.0.compiled-library" and a python setup file. Note that the version number may differ from the file you actually downloaded.

Twingrind PLC library

Start the TwinCAT XAE Shell or the Visual Studio Version you are usually using to develop TwinCAT PLCs. Then, in the menubar, select PLC and then Library Repository... (see figures below)

TwinCAT XAE ShellTwinCAT XAE ShellAdd library

In the library-repository dialog, click on Install and navigate to the file compiled-library file and select it. Then, click on Open to install the struckig library into your TwinCAT environment, and you are ready to use it.

To make the Twingrind library available to the PLC, open the solution, which contains the PLC you want to profile. In the solution explorer, expand the PLC you are interested in and right-click on References. Select Add library. In the dialog, search for Twingrind, then select the item and click on Ok

pytwingrind python-module

Open a command prompt and navigate to the pytwingrind-0.1.1-py3-none-any.whl file. Then use the following command to install the python module on your system. Make sure that your python environment is reachable in your path variable.

pip install pytwingrind-0.1.1-py3-none-any.whl

After running the command successfully, the executable twingrind.exe should be available in your path.

Preparation

Most of the profiling boilerplate code is generated by the Twingrind command prepare. However, the following lines of code have to be added manually to your PLC's MAIN program.

Twingrind.Profiler();
Twingrind.Profiler.Push(0);

// <your existing implementation comes here>

Twingrind.Profiler.Pop(0);

The PRG Profiler has inputs to control the profiler, we will later add it to the watch window to control it. The methods Push and Pop are used for time measurements. Calling this method in the implementation of your main program is mandatory. After added the 3 lines of code, as shown above, use twingrind.exe prepare to add similar code section to all methods of your PLC.

twingrind prepare -d <PATH_TO_FOLDER_CONTAINING_PLCPROJ> -m <PATH_TO_HASHMAP_FILE>

The command transverses through the entire code base located at the given directory. For all methods it adds a similar call to the profiler in the header and the footer. The method calls are identified by id's and can be converted to readable strings by a hashmap file, which is the output of prepare.py. The file that is generated by this call is needed subsequently in reconstruct.

Please make sure to use a directory containing your PLC

Using Twingrind

Activate

You can now activate your PLC on your target and work as you are used to. Note that the Profiler adds some overhead to your code. making execution a bit slower. Usually you should not notice a big impact though. To start profiling navigate to your MAIN programm, right click on Profiler and Add Watch.

Add WatchWatch

Then search for Twingrind.Profiler in the Watch panel and expand the node. You can then use the watch window to

  • Capture the callstack of a single frame of your PLC by a rising edge of CaptureOnce
  • Run Captures continuously by setting CaptureContinuous=TRUE. For this setting you can also specify a cpu time threshold such that only frames with a certain percentage-based usage of your CPU are captured.
  • The library includes a parameter MAX_FRAMES, which is used to adjust the maximum amount of recorded frames. If FrameIndex=MAX_FRAMES no new captures will be performed by the Profiler. In order to reset already taken recordings you can give a rising edge on Reset. This will internally remove all data and set FrameIndex=0 again.

Process snapshot

The process command is a shortcut for fetch and then reconstruct every snapshot that was captured. Usually this is the command that you want to work with.

twingrind process -m hashmap -t 1

Here -m hashmap refers to the hashmap that has been created for your PLC during preparation. -t 1 means that the cycletime of your PLC is 1ms. Adapt these parameters accordingly to your setup, use twingrind process -h for a detailed listing of all arguments.

Take snapshot

Run the following command to read out all data from your PLC. Note that recording has to be disabled before calling this issue. Disable continous capturing by setting CaptureContinuous=FALSE in the watch window and then call

twingrind fetch

to read all recorded frames from the PLC. The resulting data is the output of fetch and is stored in the current directory. Latter files contain base64 encoded information about the callstack and can be converted to the callgrind format by reconstruct. The fetch command per default connects to the local target and with the PLC that is running on port 851. However, the command has several arguments to control its behavior, use twingrind fetch -h for a detailed listing.

Convert snapshot

Use the following command to reconstruct a frame.

python reconstruct -m <HASHMAPFILE> -c <CALLSTACK> -t 1

Creates a callgrind file in the current directory. This script uses a previously generated hashmap (output of prepare) together with a recorded callstack (output of fetch). Run the reconstruct command for all frames that were exported by *fetch. You may then open qcachegrind to visualize the callstack of your captured cycles. The command comes with some arguments to control its behavior, for details refer to twingrind reconstruct -h

In the images below the first one shows the overview over a complete cycle. The PLC that I was running when taking this picture didn't us a lot of cpu ticks that is why there is a lot of empty space in CYCLE::CYCLE. The second image is zoomed into the MAIN PRG.

In the images below the first one shows the overview over a complete cycle. The PLC that I was running when taking this picture didn't us a lot of cpu ticks that is why there is a lot of empty space in CYCLE::CYCLE. The second image is zoomed into the MAIN PRG.

Callgrind demoCallgrind demo

Cleanup

To cleanup your code from code that was added in the Prepare section you can run the clean as follows

twingrind clean -d <PATH_TO_FOLDER_CONTAINING_PLCPROJ> 

The command transverses through the entire code base of the plc located at the given directory. For all methods, the command removes the header function call and a the footer function call to the profiler library that were previously generated by using the "prepare".

You might also like...
Create HTML profiling reports from pandas DataFrame objects
Create HTML profiling reports from pandas DataFrame objects

Pandas Profiling Documentation | Slack | Stack Overflow Generates profile reports from a pandas DataFrame. The pandas df.describe() function is great

Create HTML profiling reports from pandas DataFrame objects
Create HTML profiling reports from pandas DataFrame objects

Pandas Profiling Documentation | Slack | Stack Overflow Generates profile reports from a pandas DataFrame. The pandas df.describe() function is great

An ultra fast tiny model for lane detection, using onnx_parser, TensorRTAPI, torch2trt to accelerate. our model support for int8, dynamic input and profiling. (Nvidia-Alibaba-TensoRT-hackathon2021)

Ultra_Fast_Lane_Detection_TensorRT An ultra fast tiny model for lane detection, using onnx_parser, TensorRTAPI to accelerate. our model support for in

Silk is a live profiling and inspection tool for the Django framework.
Silk is a live profiling and inspection tool for the Django framework.

Silk is a live profiling and inspection tool for the Django framework. Silk intercepts and stores HTTP requests and database queries before presenting them in a user interface for further inspection:

Silky smooth profiling for Django
Silky smooth profiling for Django

Silk Silk is a live profiling and inspection tool for the Django framework. Silk intercepts and stores HTTP requests and database queries before prese

Silky smooth profiling for Django
Silky smooth profiling for Django

Silk Silk is a live profiling and inspection tool for the Django framework. Silk intercepts and stores HTTP requests and database queries before prese

VizTracer is a low-overhead logging/debugging/profiling tool that can trace and visualize your python code execution.
VizTracer is a low-overhead logging/debugging/profiling tool that can trace and visualize your python code execution.

VizTracer is a low-overhead logging/debugging/profiling tool that can trace and visualize your python code execution.

Intelligent Employer Profiling Platform.
Intelligent Employer Profiling Platform.

Intelligent Employer Profiling Platform Setup Instructions Generating Model Data Ensure that Python 3.9+ and pip is installed. Install project depende

pycallgraph is a Python module that creates call graphs for Python programs.
pycallgraph is a Python module that creates call graphs for Python programs.

Project Abandoned Many apologies. I've stopped maintaining this project due to personal time constraints. Blog post with more information. I'm happy t

xlwings is a BSD-licensed Python library that makes it easy to call Python from Excel and vice versa. It works with Microsoft Excel on Windows and macOS. Sign up for the newsletter or follow us on twitter via

xlwings - Make Excel fly with Python! xlwings CE xlwings CE is a BSD-licensed Python library that makes it easy to call Python from Excel and vice ver

RPyC (Remote Python Call) - A transparent and symmetric RPC library for python
RPyC (Remote Python Call) - A transparent and symmetric RPC library for python

RPyC (pronounced like are-pie-see), or Remote Python Call, is a transparent library for symmetrical remote procedure calls, clustering, and distribute

PyCG: Practical Python Call Graphs

PyCG - Practical Python Call Graphs PyCG generates call graphs for Python code using static analysis. It efficiently supports Higher order functions T

🚴 Call stack profiler for Python. Shows you why your code is slow!
🚴 Call stack profiler for Python. Shows you why your code is slow!

pyinstrument Pyinstrument is a Python profiler. A profiler is a tool to help you 'optimize' your code - make it faster. It sounds obvious, but to get

Oncall is a calendar tool designed for scheduling and managing on-call shifts. It can be used as source of dynamic ownership info for paging systems like http://iris.claims.
Oncall is a calendar tool designed for scheduling and managing on-call shifts. It can be used as source of dynamic ownership info for paging systems like http://iris.claims.

Oncall See admin docs for information on how to run and manage Oncall. Development setup Prerequisites Debian/Ubuntu - sudo apt-get install libsasl2-d

pspsps(1) is a compyuter software to call an online catgirl to the Linux terminyal.
pspsps(1) is a compyuter software to call an online catgirl to the Linux terminyal.

pspsps(1): call a catgirl from the Internyet to the Linux terminyal show processes: ps show catgirls: pspsps —@[email protected] pspsps(1) is a compyute

Stop writing scripts to interact with your APIs. Call them as CLIs instead.
Stop writing scripts to interact with your APIs. Call them as CLIs instead.

Zum Stop writing scripts to interact with your APIs. Call them as CLIs instead. Zum (German word roughly meaning "to the" or "to" depending on the con

printstack is a Python package that adds stack trace links to the builtin print function, so that editors such as PyCharm can link you to the source of the print call.
printstack is a Python package that adds stack trace links to the builtin print function, so that editors such as PyCharm can link you to the source of the print call.

printstack is a Python package that adds stack trace links to the builtin print function, so that editors such as PyCharm can link to the source of the print call.

Audio processor to map oracle notes in the VoG raid in Destiny 2 to call outs.

vog_oracles Audio processor to map oracle notes in the VoG raid in Destiny 2 to call outs. Huge thanks to mzucker on GitHub for the note detection cod

A lightweight (serverless) native python parallel processing framework based on simple decorators and call graphs.
A lightweight (serverless) native python parallel processing framework based on simple decorators and call graphs.

A lightweight (serverless) native python parallel processing framework based on simple decorators and call graphs, supporting both control flow and dataflow execution paradigms as well as de-centralized CPU & GPU scheduling.

Comments
  • Symbol not found

    Symbol not found

    Hi,

    if you compile you plc programm with the compiler definition "TC_SYM_WITH_NAMESPACE" you cant get access to the symbol. Because the Namspace of the Library will be added to the symbolname.

    I would suggesst you to add an parameter for the symbol prefix,

    Best Regards Markus

    opened by mriepl 3
  • Skip SFC implementations

    Skip SFC implementations

    If a project has one or more programs implemented in SFC the regex go into a long search loops given that SFC files can be 30k+ lines. The prepare function just holds forever.

    This just adds a check and skip for those types of files.

    opened by kdorsel 1
  • Symbol not found for Profiler.Data

    Symbol not found for Profiler.Data

    https://github.com/stefanbesler/twingrind/blob/759669c64ce397d49bce641d9e8b298a72afd2dd/pytwingrind/pytwingrind/fetch.py#L49

    Typo für symbolname "Profiler.Data[0,1]"! You have added an space (Hex 20) between the two array positions! Please delete the space in [{frame},< >{task}]

    if you have done this please release an new version, thx

    opened by mriepl 1
Releases(v0.4.0)
  • v0.4.0(Oct 6, 2022)

    • feat: Trigger profiling from the commandline. twingrind fetch and twingrind process got new arguments to directly trigger profiling (single shots and resets) from the command line. -s <NUMBER> takes <NUMBER> profiles and -r resets all data, which was already recorded. The latter happens before taking shots, which allows to use both flags to be used simultanously (i.e. twingrind fetch -rs1).
    • feat: Support for using symbol namespaces. TwinCAT could be configured to use the namespace before any symbols by setting the TC_SYM_WITH_NAMESPACE compiler define. Twingrind can now deal with this usecase automatically by trying to connect with the profiler with the default namespace. Additionally, the namespace of the Profiler can be set externally in twingrind fetch and twingrind process by a the new namespace argument -N <Namespace>.
    Source code(tar.gz)
    Source code(zip)
    pytwingrind-0.4.0-py3-none-any.whl(9.98 KB)
    Twingrind_0.4.0.0.compiled-library(20.55 KB)
  • v0.3.1(Jun 16, 2022)

  • v0.3.0(Jun 2, 2022)

    • support for profile modes for capturing only the slowest or faster cycles
    • improved implementation for continuous capturing of profiles. Rather than only capturing X number of callstacks there is now a mode to overwrite old callstacks in favor of new ones (Mode=All), the old behavior can still be activated as well (Mode=FirstOnesOnly)
    • increased the time resolution of profiling by a factor of 10 from 10us to 0.1us
    • added a parameter to pytwingrind to prefix the files that are generated by it. This facilities comparing of callstacks
    Source code(tar.gz)
    Source code(zip)
    pytwingrind-0.3.0-py3-none-any.whl(9.10 KB)
    Twingrind_0.3.0.0.compiled-library(20.54 KB)
  • v0.2.1(May 24, 2022)

  • v0.2.0(May 17, 2022)

  • v0.1.1(Apr 11, 2022)

  • v0.1.0(Apr 8, 2022)

    Initial release of twingrind, which includes the following features

    • Call-graph profiling of a TwinCAT PLC
    • Twingrind_0.1.0.0.compiled-library, which implements the PLC-side profiling mechanism
    • pytwingrind/prepare.py script to add profiling guards to a PLC and/or PLC libraries, these are used for capturing the callstack of PLC cycles
    • Twingrind.Profiler program to capture the callstack of PLC cycles, either manually or automatic with adjustable cycletime-usage threshold
    • pytwingrind/fetch.py script to extract profiling data from the PLC and store it on disk with base64 encoding
    • pytwingrind/reconstruct.py script to convert the base64 encoded file to the callgrind format

    (Known) limitations of this release

    • Only 1 task and 1 PRG are supported.
    • Profiling itself adds some overhead to your code, which can not be avoided by this method, but maybe reduced by a fair bit.
    • No unicode character support in source files
    Source code(tar.gz)
    Source code(zip)
    Twingrind_0.1.0.0.compiled-library(15.23 KB)
Owner
stefanbesler
stefanbesler
🚴 Call stack profiler for Python. Shows you why your code is slow!

pyinstrument Pyinstrument is a Python profiler. A profiler is a tool to help you 'optimize' your code - make it faster. It sounds obvious, but to get

Joe Rickerby 5k Jan 1, 2023
Digital Twin Mobility Profiling: A Spatio-Temporal Graph Learning Approach

Digital Twin Mobility Profiling: A Spatio-Temporal Graph Learning Approach This is the implementation of traffic prediction code in DTMP based on PyTo

chenxin 1 Dec 19, 2021
Source code for CIKM 2021 paper for Relation-aware Heterogeneous Graph for User Profiling

RHGN Source code for CIKM 2021 paper for Relation-aware Heterogeneous Graph for User Profiling Dependencies torch==1.6.0 torchvision==0.7.0 dgl==0.7.1

Big Data and Multi-modal Computing Group, CRIPAC 6 Nov 29, 2022
Pyan3 - Offline call graph generator for Python 3

Pyan takes one or more Python source files, performs a (rather superficial) static analysis, and constructs a directed graph of the objects in the combined source, and how they define or use each other. The graph can be output for rendering by GraphViz or yEd.

Juha Jeronen 235 Jan 2, 2023
(OLD REPO) Line-by-line profiling for Python - Current repo ->

line_profiler and kernprof line_profiler is a module for doing line-by-line profiling of functions. kernprof is a convenient script for running either

Robert Kern 3.6k Jan 6, 2023
Soda SQL Data testing, monitoring and profiling for SQL accessible data.

Soda SQL Data testing, monitoring and profiling for SQL accessible data. What does Soda SQL do? Soda SQL allows you to Stop your pipeline when bad dat

Soda Data Monitoring 51 Jan 1, 2023
Silky smooth profiling for Django

Silk Silk is a live profiling and inspection tool for the Django framework. Silk intercepts and stores HTTP requests and database queries before prese

Jazzband 3.7k Jan 4, 2023
Line-by-line profiling for Python

line_profiler and kernprof NOTICE: This is the official line_profiler repository. The most recent version of line-profiler on pypi points to this repo

OpenPyUtils 1.6k Dec 31, 2022
Create HTML profiling reports from pandas DataFrame objects

Pandas Profiling Documentation | Slack | Stack Overflow Generates profile reports from a pandas DataFrame. The pandas df.describe() function is great

null 10k Jan 1, 2023
Create HTML profiling reports from pandas DataFrame objects

Pandas Profiling Documentation | Slack | Stack Overflow Generates profile reports from a pandas DataFrame. The pandas df.describe() function is great

null 6.8k Feb 18, 2021