I still haven't had success yet with parsing the FB messages. One issues was that timestamp = pd.to_datetime(content, format='%A, %B %d, %Y at %H:%M%p', exact=False).strftime("%s") had an invalid format string. I changed it to "%S" since I assume that's what you meant. However, I now get the following issue when running analyse.py.

Traceback (most recent call last):
  File "analyse.py", line 90, in <module>
    print(plot)
  File "C:\Users\asros\Anaconda3\envs\py2\lib\site-packages\ggplot\ggplot.py", line 116, in __repr__
    self.make()
  File "C:\Users\asros\Anaconda3\envs\py2\lib\site-packages\ggplot\ggplot.py", line 641, in make
    self.apply_axis_labels()
  File "C:\Users\asros\Anaconda3\envs\py2\lib\site-packages\ggplot\ggplot.py", line 231, in apply_axis_labels
    labels.append(self.xtick_formatter(label_text))
  File "C:\Users\asros\Anaconda3\envs\py2\lib\site-packages\matplotlib\dates.py", line 542, in __call__
    dt = num2date(x, self.tz)
  File "C:\Users\asros\Anaconda3\envs\py2\lib\site-packages\matplotlib\dates.py", line 445, in num2date
    return _from_ordinalf(x, tz)
  File "C:\Users\asros\Anaconda3\envs\py2\lib\site-packages\matplotlib\dates.py", line 260, in _from_ordinalf
    dt = datetime.datetime.fromordinal(ix).replace(tzinfo=UTC)
ValueError: ordinal must be >= 1

Had this issue when running the parser and to solve is just add proper encoding on line 36 if anyone happens to face the same problem:

archive = etree.parse(filePath + "/" + filename,
                            etree.XMLParser(encoding='utf-8',
                                            ns_clean=True,
                                            recover=True))

should submit a PR?

Hey there,

So I've cloned the project a few times before, and even though I've gotten it to work, Chatistics doesn't work out of the box for me. My environment variables were already set, but LXML was giving me trouble, as where some dependencies that aren't listed in the requirements.txt. I noticed that there's a section for libxml in the README, but I just wanted to ask if you guys were taking any pull requests to add some fallbacks in case certain modules aren't already installed. Thanks!

I had it working on the 17th of January but I updated the master branch today and now conda activate chatistics fails with:

CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run

    $ conda init <SHELL_NAME>

Currently supported shells are:
  - bash
  - fish
  - tcsh
  - xonsh
  - zsh
  - powershell

See 'conda init --help' for more information and options.

IMPORTANT: You may need to close and restart your shell after running 'conda init'.

Before trying this I deleted the existing environment out of ~/anaconda/envs/ and ran conda env create -f environment.yml

I also updated to Catalina in the meantime if that makes any difference.

Any ideas as to how I can get this working?

Edited to remove env from conda activate chatistics.

Rationale

Addressing issue #12

Important

• Relies on telethon, therefore codebase was migrated to Python 3. Anyway, IMO we should develop in Python 3.
• Work with telegram requires registering your own app to get an access token
• Added logging
• Bump some requirements
• A lot of refactor and code style fixes, sorry :guardsman:

I get a problem when trying to parse the facebook messenger files. I get File "parse_messenger.py", line 76, in <module> timestamp = pd.to_datetime(content, format='%A, %B %d, %Y at %H:%M%p', exact=False).strftime("%s")

and

'ValueError: time data 'Sunday, 11 March 2012 at 15:11 UTC+01' does not match format '%A, %B %d, %Y at %H:%M%p' (search)

Is this perhaps due to different time stamps formats or that they are older?

infer_datetime_regex withing whatsapp.py handles single digit month day values in the whatsapp timestamp incorrectly. Causing the parser to ignore a huge amount of messages. This error depends on the first time value encountered.

I'm not quite sure why the regex isn't a constant, but this pull request should fix the problem while maintaining the flexibility of the original function.

Before: After:

Having ModuleNotFoundError while running the command

python parsers/messenger.py --own-name "John Doe"

There's __ init __.py in parsers subdirectory, still facing this issue.

Traceback (most recent call last): File "parsers/messenger.py", line 11, in <module> from parsers import log ModuleNotFoundError: No module named 'parsers'

I am having the following issue with Pandas trying to read the datetime. Do you know why it's having trouble reading the full datetime?

Traceback

105 Person Name (group? False )
Traceback (most recent call last):
  File "parse_messenger.py", line 71, in <module>
    timestamp = pd.to_datetime(content[:-7], format='%A, %B %d, %Y at %H:%M%p').strftime("%s")
  File "C:\Users\asros\Anaconda3\envs\py2\lib\site-packages\pandas\core\tools\datetimes.py", line 382, in to_datetime
    result = _convert_listlike(np.array([arg]), box, format)[0]
  File "C:\Users\asros\Anaconda3\envs\py2\lib\site-packages\pandas\core\tools\datetimes.py", line 306, in _convert_listlike
    raise e
ValueError: time data 'Monday, January 22, 2018 at 6:4' does not match format '%A, %B %d, %Y at %H:%M%p' (match)

Text:

</style><title>Conversation with Friend Name</title></head><body><a href="html/messages.htm">Back</a><br /><br /><div class="thread"><h3>Conversation with Friend Name</h3>Participants: Friend Name<div class="message"><div class="message_header"><span class="user">Friend Name</span><span class="meta">Monday, January 22, 2018 at 6:46pm CST</span></div></div><p>Oh wow</p><div class="message"><div class="message_header"><span class="user">My Name</span><span class="meta">Monday, January 22, 2018 at 6:44pm CST</span></div></div><p>blah blah blah message was here</p><div class="message"><div class="message_header"><span class="user">Friend Name</span><span class="meta">Monday, January 22, 2018 at 6:40pm CST</span></div></div><p>

I have exported the WhatsApp chat from a single conversation, and after running python parse.py whatsapp --own-name "firstname lastname" each row in the "senderName" column has a prefix of AM/PM (from the timestamp), e.g. "AM - firstname lastname" or "PM - firstname lastname". This leads to errors in visualization, etc.

Here are a few lines from the WhatsApp export for reproducibility:

9/18/18, 2:45 PM - firstname1 lastname1: I'm sorry! I was talking to people throughout the lunch break! 9/18/18, 4:24 PM - firstname2 lastname2: I'm glad it is going so well! 9/18/18, 4:25 PM - firstname2 lastname2: Here's a quick outtake from the filming 9/18/18, 4:25 PM - firstname2 lastname2: <Media omitted>

I followed the steps outlined in the readme.

Downloaded messages from facebook in JSON format. Unzipped and put the content from the messages folder in my raw_data/messenger directory.

When I run python parse.py messenger I get this error:

2020-01-23 04:27:34,164 [INFO ] [parsers.mess]: Parsing Facebook messenger data...
2020-01-23 04:27:34,165 [ERROR] [parsers.mess]: No input files found under raw_data/messenger

I mistakenly exported my Facebook messenger data content to the messenger folder, instead of the entire messages folder i had unzipped from the Facebook download file. I saw in an other thread, that others had also done this. The readme could be changed so this would be avoided.

If i run export.py -format pkl then i get an error saying:

Traceback (most recent call last):
  File "export.py", line 59, in <module>
    main()
  File "export.py", line 52, in main
    with open(f_name, 'wb', encoding="utf8") as f:
ValueError: binary mode doesn't take an encoding argument

The error can be avoided by not setting encoding in the open() function.

2020-09-26 19:38:57,867 [INFO ] [matplotlib.f]: Generating new fontManager, this may take some time... 2020-09-26 19:39:01,947 [INFO ] [parsers.mess]: Parsing Facebook messenger data... 2020-09-26 19:39:01,948 [INFO ] [parsers.mess]: Trying to infer own_name from data... 2020-09-26 19:39:02,037 [INFO ] [parsers.mess]: Successfully inferred own-name to be User1 2020-09-26 19:39:02,061 [INFO ] [parsers.mess]: Group chats are not supported yet. 2020-09-26 19:39:02,079 [INFO ] [parsers.mess]: Group chats are not supported yet. 2020-09-26 19:39:02,100 [INFO ] [parsers.mess]: Group chats are not supported yet. 2020-09-26 19:39:02,109 [INFO ] [parsers.mess]: 0 messages parsed. 2020-09-26 19:39:02,109 [INFO ] [parsers.mess]: Nothing to save.

Can I kindly ask if there will be group chat support in the near future ?

@teamfridge not sure I understand. When you parse logs from a group chat, each message should have its own senderName, which allows you to perform analysis on a per-individual basis. You just have to group/filter the dataframes using this field.

Originally posted by @MasterScrat in https://github.com/MasterScrat/Chatistics/issues/44#issuecomment-577400599

Hi,

I tried:

$ python parse.py hangouts -f path/to/my/hangouts.json
2020-07-29 04:48:07,372 [INFO ] [parsers.hang]: Parsing Google Hangouts data...
2020-07-29 04:48:07,372 [INFO ] [parsers.hang]: Reading archive file path/to/my/hangouts.json...
2020-07-29 04:48:09,348 [INFO ] [parsers.hang]: Trying to infer own_name from data...
Traceback (most recent call last):
  File "parse.py", line 84, in <module>
    ArgParse()
  File "parse.py", line 42, in __init__
    getattr(self, args.command)()
  File "parse.py", line 60, in hangouts
    main(args.own_name, args.file_path, args.max)
  File "/path/to/Chatistics/parsers/hangouts.py", line 21, in main
    own_name = infer_own_name(archive)
  File "/path/to/Chatistics/parsers/hangouts.py", line 167, in infer_own_name
    for conversation in archive['conversations']:
KeyError: 'conversations'

Then, I tried:

$ python parse.py hangouts --own-name MARCO -f path/to/my/hangouts.json
2020-07-29 04:49:05,521 [INFO ] [parsers.hang]: Parsing Google Hangouts data...
2020-07-29 04:49:05,521 [INFO ] [parsers.hang]: Reading archive file path/to/my/hangouts.json...
2020-07-29 04:49:07,308 [INFO ] [parsers.hang]: Extracting messages...
Traceback (most recent call last):
  File "parse.py", line 84, in <module>
    ArgParse()
  File "parse.py", line 42, in __init__
    getattr(self, args.command)()
  File "parse.py", line 60, in hangouts
    main(args.own_name, args.file_path, args.max)
  File "/path/to/Chatistics/parsers/hangouts.py", line 22, in main
    data = parse_messages(archive, own_name)
  File "/path/to/Chatistics/parsers/hangouts.py", line 102, in parse_messages
    for conversation in archive['conversations']:
KeyError: 'conversations'

Then I tried to copy the file in ./raw_data/hangouts/, call python parse.py hangouts --own-name MARCO and got the same error.

Hi Florian! Good repo, I had a lot of fun with it :)

I have added stopwords files for some missing languages. They were extracted from here and converted using a Python script.

New languages included: arabic, bulgarian, catalan, czech, danish, dutch, finnish, german, hebrew, hindi, hungarian, indonesian, malaysian, italian, norwegian, polish, portuguese, romanian, russian, slovak, spanish, swedish, turkish, ukrainian and vietnamese

Hello everyone!

I start saying I'm very inexperienced with... well... everything.

While parsing from WhatsApp works smoothly, I'm having some troubles with messenger chats.

This is the output I'm having:

2020-04-25 23:31:17,481 [INFO ] [parsers.mess]: Parsing Facebook messenger data...
Traceback (most recent call last):
  File "parse.py", line 84, in <module>
    ArgParse()
  File "parse.py", line 42, in __init__
    getattr(self, args.command)()
  File "parse.py", line 69, in messenger
    main(args.own_name, args.file_path, args.max)
  File "/Users/*****/Chatistics/parsers/messenger.py", line 22, in main
    data = parse_messages(file_path, own_name)
  File "/Users/*****/Chatistics/parsers/messenger.py", line 64, in parse_messages
    content = fix_text_encoding(content)
  File "/Users/*****/Chatistics/parsers/messenger.py", line 79, in fix_text_encoding
    return text.encode('latin1').decode('utf8')
UnicodeEncodeError: 'latin-1' codec can't encode character '\u201c' in position 17: ordinal not in range(256)

The output from export then prints:

...
2020-04-26 00:43:32,913 [INFO ] [utils       ]: Could not find any data for platform messenger

In /raw_data/messenger I have all the conversations folders that were previously contained in the inbox folder in the original archive downloaded from Facebook. Like this: /raw_data/messenger/username/message_1.json. (I think the documentation should be updated, it makes it look like that the inbox folder needs to be there too).

I'm working in Jupiter lab.

Thanks in advance!