现在获取资源的链接一般是 https://yoursite.com/https://targetsite.com/resource/path/

这样一来会有两个问题：

• 丑，资源请求路径太长，不好看。
• 直接暴露源站。

提出一个设想：

是否可以在 Meta 中增加一个 alias 作为源站 base_url 的替代或标识，并且作为一级资源路径插入到路由中，如 https://yoursite.com/<alias>/resource/path/，这样一来既可以满足区分多站点的需求，又可以解决上面提到的两个问题。

在官方仓库中的例子（查看源码）有利用 flask 的路由进行自定义路由的实现，如果有多个站点多个请求路径，这样写在 items 里有一份路由，在这里面又要再写一份路由，显得有点机械了。

Thanks.

In working with toapi I came across a scenario where the web page had an HTML table that was paginated.

Clicking on "next page" would issue an ajax post request to fetch the next set of records in the data set.

Is there anyway to accomplish this with toapi?

1. 如果我想解析的一个页面是通过post请求才能得到的

请问 toapi 提供这样的方式么？ 我看在定义settings是有一个参数是ajax=true 那么我发送ajax请求的data应该定义在哪里呢？ 翻了一圈文档和issues都没找到

2. 关于items的编写

自带的XPath方法返回的好像是处理过的值而不是一个 etree element 这样我比如想要获取h1下所有的文本（包括子标签）就不可以用string(.)方法 必须得再写一个clean_xx的方法

另外 能否加入bs4的支持呢？

最后希望该项目能越做越好！ 真的很棒！

python版本是3.5 toapi版本0.2.2

toapi new api
cd api
toapi run

执行topic run时报错

➜  api toapi run
Traceback (most recent call last):
  File "/usr/local/bin/toapi", line 9, in <module>
    load_entry_point('toapi==0.2.2', 'console_scripts', 'toapi')()
  File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/toapi/cli.py", line 81, in run
    app = importlib.import_module('app', base_path)
  File "/usr/lib/python3.5/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 986, in _gcd_import
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 665, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/zz/code/python/toapi_project/api/app.py", line 7, in <module>
    api.register(Page)
  File "/usr/local/lib/python3.5/dist-packages/toapi/api.py", line 31, in register
    item.__pattern__ = re.compile(item.__base_url__ + item.Meta.route)
TypeError: Can't convert 'dict' object to str implicitly

item.__base_url__是str, item.Meta.route是字典

Good evening,

I have a question regarding the cache and its time to live.

Let's say I want to turn some site into an API and want the results of the very first request to be cached for one hour. How would I specify that in the settings? Is such a setup even possible?

I tried setting ttl: 60 * 60, assuming that that would do the trick. But to me it seems it doesn't...

Could you please clarify?

Thanks in advance.

Hello, I am relatively new to python web development. And while I am mainly working on a mobile app. I found topapi to be a perfect companion for my backend requirements. I am now almost ready to launch my app, but am struggling to find a good production hosting environment for the toapi server code. Mainly looking around using heroku or aws or google app engine for hosting server.

I was wondering if you can provide some instructions for deploying to a production quality server. I did go over this deploy link but still not able to link the content to the actual toapi codebase.

And advise on how can I move forward with this.

Thank you again,

python 3.7 toapi 2.1.1

Traceback (most recent call last):
  File "main.py", line 5, in <module>
    api = Api()
  File "/usr/local/lib/python3.7/site-packages/toapi/api.py", line 24, in __init__
    self.__init_server()
  File "/usr/local/lib/python3.7/site-packages/toapi/api.py", line 27, in __init_server
    self.app.logger.setLevel(logging.ERROR)
AttributeError: module 'flask.logging' has no attribute 'ERROR'

[root@test python3]# python --version Python 3.6.2

[root@test python3]# toapi new toapi/toapi-pic Usage: toapi [OPTIONS] COMMAND [ARGS]...

Error: No such command "new".

help ?

Traceback (most recent call last): File "app.py", line 2, in from htmlparsing import Attr, Text File "/usr/local/lib/python2.7/dist-packages/htmlparsing-0.1.5-py2.7.egg/htmlparsing.py", line 21 def init(self, text: str): ^ SyntaxError: invalid syntax

Hello, I need to get access to the raw HTML in one of Item instances. Currently the XPath or CSS selectors always convert the node as a string. But in my use case once I select certain part of my webpage, I need to do some post-processing in my clean_ method. But I can only get a string passed into it. Is there a way to get a rawHTML passed into my clean_ method for a given key.

Thank you,

class Meta: source = NONE route = {'/search/:id': '/search/:id'}

Right now ID for host url passes directly into source url. Is there a way we can modify ID before passing them on?

For example, I need to map the query 127.0.0.1:5000/search/1 to bing.com/search/100

So I am going to have to multiply :id with 100 before passing it as argument. Not sure if that makes sense.

Bumps ujson from 4.0.2 to 5.4.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

When I ran this example, I reported the following error

2019/09/19 10:15:50 [Register] OK <Post: /posts /news?p=1> 2019/09/19 10:15:50 [Register] OK <Post: /posts?page={page} /news?p={page}> 2019/09/19 10:15:50 [Register] OK <Page: /posts /news?p=1> 2019/09/19 10:15:50 [Register] OK <Page: /posts?page={page} /news?p={page}> 2019/09/19 10:15:50 [Serving ] OK http://0.0.0.0:5001 2019/09/19 10:15:50 [Serving ] FAIL Windows error 1 2019/09/19 10:15:50 [Serving ] FAIL Traceback (most recent call last): File "D:\python\lib\site-packages\toapi\api.py", line 50, in run self.app.run(host, port, **options) File "D:\python\lib\site-packages\flask\app.py", line 938, in run cli.show_server_banner(self.env, self.debug, self.name, False) File "D:\python\lib\site-packages\flask\cli.py", line 629, in show_server_banner click.echo(message) File "D:\python\lib\site-packages\click\utils.py", line 260, in echo file.write(message) File "D:\python\lib\site-packages\click_winconsole.py", line 180, in write return self._text_stream.write(x) File "D:\python\lib\site-packages\click_winconsole.py", line 164, in write raise OSError(self._get_error_message(GetLastError())) OSError: Windows error 1

toapi, version 2.1.0 Flask 1.0.2 Python 3.6.0

I use:

class ProductPage(Item):
      coupon = Attr('.coupon', 'title')

However some product pages do not contain the coupon html so they fail with

  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/htmlparsing.py", line 79, in parse
    return element.css(self.selector)[0].attrs[self.attr]
IndexError: list index out of range

What's the best practice to deal with that situation?