Robot Framework 5.0 survey results
We had a survey asking what features Robot Framework community members would like to see in the forthcoming Robot Framework 5.0 release. In addition to that, respondents could give free comments to the Robot Framework development team and to the Robot Framework Foundation that sponsors Robot Framework development.
Voted issues are listed by priority in the section below and all responses to the survey are after them. Also raw data is available with all personal information removed.
Two online tickets to RoboCon 2022 were raffled among the respondents who left their contact information. The lucky winners, selected by the random module, are Anton Pääkkönen and Veijo Yli-Kätkä. Congratulations to them and thanks for everyone who participated the survey!
Issues by priority
Voted issues are listed below by priority with comments explaining why they were considered important. Issues were voted in three priorities and higher priority issues had a higher weight. Comments are got only from the highest priority issues. All issues currently targeted for RF 5.0 can be found in the issue tracker.
- #3075 Native support for
TRY/EXCEPTfunctionality (weighted priority: 86)- "Errors" are part of the business when working in process automation. Exceptions are ok and require handling. Which is rather difficult in RF right now.
- Native TRY / EXCEPT is important for implementing RPA error handling workflows. Other control structures, such as WHILE, are also highly requested among RPA users (who wants to use FOR in range 0..999?) but out of everything other TRY / EXCEPT is difficult to implement with current constructs. Basically every RPA bot will benefit from this.
- The current way to handle errors is really clunky. Try/except is a well-known pattern in many languages.
- In RPA cases exceptions are common and need to be handled with proper code paths
- Many operations in RF can raise exceptions and try/except enables much better syntax for handling those than what is currently available.
- This brings large part of funcionality higher, to robot
- Native support for try except can help to handle exceptions well and prevent unwanted failures.
- handle exceptions is a must during automation developing
- Better error control for actions we know that may fail, and then run a workaround or repeat attempt.
- Having the ability to use a try catch code block would help clean up a lot of code, along with solve a number of problems
- This would simplify keywords where we want to do the same thing with both error inducing test data and successful test data
- #4084
WHILEloop (weighted priority: 85)- While loop support is probably one of the most common questions I get from our engineers as they start with Robot.
- Make polling solutions more readable
- Because of a variety of use cases to build around a WHILE-loop.
- It's missing at the moment and for loop has to be used as a workaround
- I have multiple cases where I need to execute my API operation tests with an 'infinite' loop which is only supposed to be terminated in case there is a change in e.g. a variable's value etc.
Since 'real' While loops are not available, I use a For loop as a workaround. Nevertheless, due to its nature, this loop is finite and will terminate once the iteration counter has reached its maximum value. - Have got a lot of cases when it would be more useful instead WUKS
- More flexible code! And we have many polling situations in our product so it will be very helpful!
- It'll be quite handy and useful in certain situations instead of using WUKS.
- #4078 New
RETURNstatement for returning from a user keyword (weighted priority: 60)- I really like to use the output from a keyword as an argument in another keyword. Without hassling with scopes like test/suite or global.
- #4093 is the most important, however, it will surely be there. :) #4078 adds a construction known from other language, which makes it possible to remove one configuration option and 4 keywords. Great!
- Sometimes I want to keep IF-return-ELSE-return logic in robot-file instead of "hiding" it away in a python-file. This would really simplify tests and make them readable to more people.
- There has been many times a situation in my work where returning value inside IF..ELSE could have been very useful.
- #4093 Inline
IFsupport (weighted priority: 43)- There have been times where I would have preferred a ternary type operation. With some work the Set Variable If keyword can be used to determine keyword execution, but with an in-line IF with an ELSE would definitely eliminate the work around.
- #4079 New
BREAKandCONTINUEstatemens forFORloop control (weighted priority: 43) - #3187 Inline keyword call syntax (weighted priority: 30)
- because the test will be more readable, less variables "spamming"
- #3423 Possibility to use output.json in addition to output.xml (weighted priority: 27)
- It would help in customization of reports
- managing results is bit tedious job
- I believe that implementing this feature greatly reduces the weight of the outputs by speeding up their processing
- #3761 Native syntax to create local variables (weighted priority: 25)
- This is one of the most weird things when writing Robot tests. Would be great to have this supported natively.
- #3278 Variable type conversion in test data (weighted priority: 15)
- #4088 Ability to register custom converters for keyword arguments (weighted priority: 14)
- #3625 Support custom statuses (weighted priority: 14)
- #3993 Provide API to get the variables from a Python file (weighted priority: 12)
- Most probably not the only person who uses RF with Python. So the tighter the integration we have the better. Would love to help out, however I am by know means a developer.
- This would open interesting doors in robot test automation.
- #2581 Scope of keywords called in resource files is globally reinterpreted when resource file is imported (weighted priority: 12)
- It is missed thing for RF to be a good language.
Keyword support in large projects becomes real headache, name conflicts, unexpected keyword usage, mess of imports
- It is missed thing for RF to be a good language.
- #3457 Remove Python 2 and Python 3.5 support (weighted priority: 11)
- The original sunset target of 2015 was announced back in 2008. That's a decade and then some. Good riddance, Python 2! Hello f-strings, path-like objects, and other cool kids!
- I want asyncio support which requires dropping python 2 support :)
- #3017 Add return type to libdoc output (weighted priority: 10)
- #4096 Multilanguage support for markers used in data (weighted priority: 9)
- This will help for sharing information
- #4020 Remove built-in Tidy tool in favor of external Robotidy (weighted priority: 9)
- #4095 Add type formatter to Log keyword (weighted priority: 7)
- #2970 Imported resource files variables not isolated (weighted priority: 6)
- #519 Given/When/Then should support other languages than English (weighted priority: 5)
- #4039 Incomplete Exception Trace in log.html (Exception Chaining) (weighted priority: 5)
- #2982 xUnit outputs: Add separate
- #3410 Add
--maxassignlengthto control how much to automatically log when assigning variables (weighted priority: 3) - #3902 Support serializing executable test suite into JSON (weighted priority: 3)
- #4089 support async keywords (weighted priority: 1)
- #4068 Make test and suite messages available as plain text and HTML separately (weighted priority: 1)
All responses
Hover mouse over the issue id to see its title.
What is the single most important feature for you? Why?
| Issue | Comment |
|---|---|
| #3075 | "Errors" are part of the business when working in process automation. Exceptions are ok and require handling. Which is rather difficult in RF right now. |
| #3075 | Native TRY / EXCEPT is important for implementing RPA error handling workflows. Other control structures, such as WHILE, are also highly requested among RPA users (who wants to use FOR in range 0..999?) but out of everything other TRY / EXCEPT is difficult to implement with current constructs. Basically every RPA bot will benefit from this. |
| #3075 | The current way to handle errors is really clunky. Try/except is a well-known pattern in many languages. |
| #3075 | In RPA cases exceptions are common and need to be handled with proper code paths |
| #3075 | Many operations in RF can raise exceptions and try/except enables much better syntax for handling those than what is currently available. |
| #2457 | Dummy |
| #3423 | It would help in customization of reports |
| #3993 | Most probably not the only person who uses RF with Python. So the tighter the integration we have the better. Would love to help out, however I am by know means a developer. |
| #4084 | While loop support is probably one of the most common questions I get from our engineers as they start with Robot. |
| #4084 | Make polling solutions more readable |
| #3075 | This brings large part of funcionality higher, to robot |
| #3075 | Native support for try except can help to handle exceptions well and prevent unwanted failures. |
| #3075 | handle exceptions is a must during automation developing |
| #3820 | People struggle with running same tests with different configs. |
| #4093 | There have been times where I would have preferred a ternary type operation. With some work the Set Variable If keyword can be used to determine keyword execution, but with an in-line IF with an ELSE would definitely eliminate the work around. |
| #3761 | This is one of the most weird things when writing Robot tests. Would be great to have this supported natively. |
| #4084 | Because of a variety of use cases to build around a WHILE-loop. |
| #3075 | Better error control for actions we know that may fail, and then run a workaround or repeat attempt. |
| #3423 | managing results is bit tedious job |
| #4084 | It's missing at the moment and for loop has to be used as a workaround |
| #4084 | I have multiple cases where I need to execute my API operation tests with an 'infinite' loop which is only supposed to be terminated in case there is a change in e.g. a variable's value etc. Since 'real' While loops are not available, I use a For loop as a workaround. Nevertheless, due to its nature, this loop is finite and will terminate once the iteration counter has reached its maximum value. |
| #4110 | This would really optimize the number of hours a big test cycle takes. |
| #2581 | It is missed thing for RF to be a good language. Keyword support in large projects becomes real headache, name conflicts, unexpected keyword usage, mess of imports |
| #4078 | I really like to use the output from a keyword as an argument in another keyword. Without hassling with scopes like test/suite or global. |
| #3457 | The original sunset target of 2015 was announced back in 2008. That's a decade and then some. Good riddance, Python 2! Hello f-strings, path-like objects, and other cool kids! |
| #4084 | Have got a lot of cases when it would be more useful instead WUKS |
| #4078 | #4093 is the most important, however, it will surely be there. :) #4078 adds a construction known from other language, which makes it possible to remove one configuration option and 4 keywords. Great! |
| #3993 | This would open interesting doors in robot test automation. |
| #3075 | Having the ability to use a try catch code block would help clean up a lot of code, along with solve a number of problems |
| #3075 | This would simplify keywords where we want to do the same thing with both error inducing test data and successful test data |
| #3457 | I want asyncio support which requires dropping python 2 support :) |
| #4078 | Sometimes I want to keep IF-return-ELSE-return logic in robot-file instead of "hiding" it away in a python-file. This would really simplify tests and make them readable to more people. |
| #3423 | I believe that implementing this feature greatly reduces the weight of the outputs by speeding up their processing |
| #3075 | |
| #4096 | This will help for sharing information |
| #4088 | |
| #2581 | |
| #3187 | because the test will be more readable, less variables "spamming" |
| #4084 | More flexible code! And we have many polling situations in our product so it will be very helpful! |
| #4078 | There has been many times a situation in my work where returning value inside IF..ELSE could have been very useful. |
| #4084 | It'll be quite handy and useful in certain situations instead of using WUKS. |
Which three issues have second-highest priority? Why?
| Issue | Comment |
|---|---|
| #2982, #3761, #3187 | They would resolve a big amount of extra-steps I am find myself too comfortable with. |
| #4078, #4084, #3423 | Better support for RETURN will simplify tasks a lot and allow for common coding patterns. Something that people just expect to work. Although inline keyword call (#3187 ) is also to be expected to just work, but that's another topic. Native WHILE will eliminate the need for ugly hacks around FOR. This is something that's not obvious for many users and it's kind of embarrassing to explain the workaround. Native support for JSON output is way overdue. XML based formats are a thing of the past and while it's easy to post process the output to JSON instead of XML, not having the support in RF core prevents the ecosystem from growing on top of a common schema. |
| #4084, #4078, #4079 | Having constructs that are familiar with "normal" programming languages makes Robot Framework easier to learn and use (since things are somewhat familiar). |
| #4078, #4084, #4079 | All the above provide elegant and natural changes into RF syntax (#3187 and #3761 are not yet ready for implementation. Their syntax proposals are still not good enough and might cause more confusion than clarify things) |
| #3187, #3278, #3017 | Robot Framework code is often at it's most clearest when types are handled automatically and user does not need to use seemingly obscure syntax like ${1} just to give a number. |
| #4088 | Dummy |
| #4078, #3017 | This would help in customization of robot framework based on the need, I have also raised #3017 sometime back |
| #4078, #4084, #4079 | Syntax and the ability to make it easier to read and learn when coming from or to other languages. |
| #4093, #4079, #3075 | These issues extend Robotframework functionality that would be used by more traditional programmers. This would help drive adoption of engineers that are looking for this kind of functionality. |
| #4079, #4093, #4078 | Making the code more readable |
| #4078, #4084, #4020 | These two bring some flow control funcionalities to robot. Robotidy is important for me as I find it very useful and with that change it might get wider audience and community support for further development. |
| #3187, #4088, #3423 | Inline keyword call would be very handy feature as it would help to reduce some steps from the code. Custom convertors would be very useful since its about data type/format. Data is the key thing for any test case so would be benefit all. Output.json would be easy to parse and use in custom dashboards. |
| #4084, #4079, #4078 | overal semantic |
| #4084, #4020, #3187 | Adding WHILE seems like the next logical item after adding the IF/ELSE, the external tidy tool is superior to the built in- I wish the external tool could be brought into the standard RF package. The ability to chain keywords could potentially be abused (limit the number of chained keywords?)- I could remove so many unnecessary local variables if I could chain some of the keywords. Super useful if you are manipulating variable values. |
| #3075 | Same as #1 :-) |
| #4079, #4020, #4093 | Inline IF feels like something I'd want to use with IFs already now, and I'd like to have it in before I feel like there's a lot to refactor in the future where I'd favor that design. Tidy is outdated, so fragmenting the tool selection is not good for the ecosystem. Break and Continue are good tools to have in the box, but not critical. I also like Try/Except and understand the RPA-importance, but I don't work with RPA so won't favor that over Break/Continue. |
| #519, #4096 | Better internationalization, and simplification of some steps with inline support. |
| #3075, #4093, #3187 | Inline keywords actually saves time and code |
| #3625, #4093, #3075 | They impact the general ease of use, code readability and integration |
| #3075, #3410, #3761 | |
| #3625 | It would be really convenient and flexible. |
| #4093, #3075, #4079 | Aligned the code with python at least, makes easy to develop tests for programmers |
| #4093, #3278, #3017 | #4093 :The inline IF removes the END #3278 : we made a keyword to see what type of variable we get. With this one we can assign the type #3017 : It is nice to see what is returned. |
| #3902, #4039, #3278 | |
| #4079, #4078, #3423 | |
| #4084, #3761, #4079 | They simplifies common constructions and/or makes them more pretty |
| #4077 | |
| #3423, #3278, #3761 | Json as the output would help with parsing the output file. The other two, in my opinion, can help greatly with making the tests/keywords more readable |
| #4078, #4079, #4084 | They provide support for common syntax patterns applicable to a wide range of use cases |
| #3278, #4088, #3761 | These would help make tests more readable and make it more clear what kind of variables should and will be moving between keywords. All these would also make tests more compact and reduce redundant ${} and other expressions |
| #4093, #4079, #4084 | They allow you to manage conditions more easily by making the writing process faster and the reading process easier |
| #3625, #4084, #3187 | |
| #4093, #4095, #4084 | Some of these are needed to improve testing |
| #3075, #4095, #4084 | |
| #2970, #4093, #4078 | |
| #3761, #4084, #4093 | to have test more readable and avoid variables "spamming" |
| #3187, #4078, #3625 | More readable code, return status will be more generic and clear now with the if-else, and custom statuses will help a lot to distinguish failures and skipped tests! |
| #4084, #4079, #3075 | They move RF closer to native coding conventions which is good. |
| #4079, #4078, #4093 |
Which other issues you would like to see included? Why?
| Issue | Comment |
|---|---|
| #3187 | This is something that a new user would expect to "just work" but the syntax proposals in that issue are not great. Still worth exploring more. |
| #519 | I see more and more non-English speakers working with Robot Framework and this may help the barrier of entry for them. |
| #4079, #3187 | |
| #4095, #519, #3625 | #4095 - helpful for debugging #519 - Would tempt cucumber users to migrate on Robot Framework. #3625 - May help some users to follow organisations standards for test case status. |
| #3075, #3457 | Good for the framework itself in gaining more userbase who respect modern support. |
| #3187, #3993, #4078 | Good to have |
| #2970 | This issue is pretty similar with the most important for me, scope of resource keywords |
| #3761, #3993 | |
| #2581, #2970 | |
| #3075, #4096 | I am a fan of writing tests in my language. |
| #3761 | Just so common that it would be nice to have |
| #4089 | async is increasingly important! |
| #4093 | Whatever makes RF more compact. |
| #3187 | It would be great to use keywords in places where you would normally only have simple strings (or variables). |
| #4068 | To improve reports |
| #2581, #2970 | |
| #3017, #4039 | #3017 to get easily the signature of a kwd, #4039 to improve analysis |
| #4093 | More readable code and reduce of testcase body length. of course sometimes having a clear block of if-else keywords is better, but especially for one liners its very good. |
| #4039, #3625 | More detailed log is always useful. Custom status could be very practical and informative for other co-workers also. |
Free comments to the development team
- #3187 and #3761 are excellent ideas, but not yet ready for implementation. Proposed syntaxes might still cause too much confusion.
- Awesome tool, keep up the good work
- Keep up the great work!
- Praiseworthy efforts on the project. Thanks for this open source automation tool
- GL. These votes are great for the community, as otherwise many of us may not have the opportunity to join the development work.
- First of all, Thank you for creating this great tool. Hats off to you guys (y)
Need below functionality as well:
All assertions should have option to return True/False instead of marking test as Pass/Fail (Should be, Should not be) - You're doing great job guys :)
- Continue to make the RF future releases more solid and strong
- Godspeed with this new release. I am very gratefull for yout hard work.
- Good luck! :)
- Thank you for your request of feedback!
- This was a good effort!
- Thank you! Keeping an eye on this process and hoping to contribute in the future
- Thanks for your good work!
- Keep doing the good work!
- Thanks for the work you do!
- Thanks for the hard work
- Keep up the good work! You are always there to help out in slack and mailing list. Hope this year i can dedicate some work time as well to help out.
Free comments to Robot Framework Foundation
- Thank you for making this available to all and 7keeping it updated on a regular basis.
- Keep up the great work!
- Thanks a lot for supporting and funding this awesome automation tool.
- Has there been any thought to developing an official style guide?
- Thanks for releasing this as open source, please continue to do so :P
- very useful and perfect ATDD
- You are on the right side
- Keep up the good work!
- Good luck! :)
- Thanks for all the fish.
- Thanks for robot framework!
- It will be good to have some students (College) activities
- I am sure the foundation will grow year by year and it definitely deserves it! Really good work with the conferences, well done!
- Foundation could arrange more trainings and workshops than just once/twice a year.
4 Sep 10, 2022
7 Oct 9, 2022
1 Nov 30, 2021
1 Dec 6, 2021
1 Dec 31, 2021
1 Aug 16, 2022
91 Dec 9, 2022
2 Feb 26, 2022
3 Aug 16, 2022
1 Oct 21, 2021
9 Nov 8, 2021
4 Dec 6, 2021
1 Nov 4, 2021
1 Nov 10, 2021
3 Oct 24, 2022
5 Nov 29, 2021
2 Oct 31, 2021
8 Oct 24, 2022
1 Nov 18, 2021