Merge branch 'devel'

Author: or-else

INSTALL.md

@@ -127,11 +127,11 @@ MongoDB should run as single node replicaset. See https://docs.mongodb.com/manua
 
 2. Run DB initializer
 	```
-	$GOPATH/bin/init-db -config=./tinode-db/tinode.conf
+	$GOPATH/bin/tinode-db -config=./tinode-db/tinode.conf
 	```
 	add `-data=./tinode-db/data.json` flag if you want sample data to be loaded:
 	```
-	$GOPATH/bin/init-db -config=./tinode-db/tinode.conf -data=./tinode-db/data.json
+	$GOPATH/bin/tinode-db -config=./tinode-db/tinode.conf -data=./tinode-db/data.json
 	```
 
 	DB initializer needs to be run only once per installation. See [instructions](tinode-db/README.md) for more options.
@@ -145,7 +145,7 @@ MongoDB should run as single node replicaset. See https://docs.mongodb.com/manua
 
 5. Run the server
 	```
-	$GOPATH/bin/tinode -config=./server/tinode.conf -static_data=$HOME/tinode/webapp/
+	$GOPATH/bin/server -config=./server/tinode.conf -static_data=$HOME/tinode/webapp/
 	```
 
 6. Test your installation by pointing your browser to [http://localhost:6060/](http://localhost:6060/). The static files from the `-static_data` path are served at web root `/`. You can change this by editing the line `static_mount` in the config file.

README.md

@@ -109,14 +109,14 @@ When you register a new account you are asked for an email address to send valid
   * JSON or [protobuf version 3](https://developers.google.com/protocol-buffers/) wire protocols.
   * Bindings for various programming languages:
     * Javascript with no external dependencies.
-    * Java with dependencies on [Jackson](https://github.com/FasterXML/jackson) and [Java-Websocket](https://github.com/TooTallNate/Java-WebSocket). Suitable for Android but with no Android SDK dependencies.
+    * Java with dependencies on [Jackson](https://github.com/FasterXML/jackson), [Java-Websocket](https://github.com/TooTallNate/Java-WebSocket), [ICU4J](https://github.com/unicode-org/icu). Suitable for Android but with no Android SDK dependencies.
     * Swift with no external dependencies.
     * C/C++, C#, Go, Python, PHP, Ruby and many other languages using [gRPC](https://grpc.io/docs/languages/).
-  * Choice of a database backend. Other databases can be added with by writing [adapters](server/db/adapter.go).
-    * MySQL
+  * Choice of a database backend. Other databases can be added by writing [adapters](server/db/adapter.go).
+    * MySQL (and MariaDB, Percona as long as they remain SQL and wire protocol compatible)
     * PostgreSQL
     * MongoDB
-    * [RethinkDB](http://rethinkdb.com/)
+    * [RethinkDB](http://rethinkdb.com/). Support is deprecated because RethinkDB is no longer being developed
 
 ### Planned
 

README_ko.md

@@ -19,7 +19,7 @@ Tinode는 XMPP/ Jabber 가 아닙니다. Tinode는 XMPP와 호환되지 않습
 
 XMPP: XML에 기반한 메시지 지향 통신 프로토콜
 
-IM: Instant Messanger
+IM: Instant Messenger
 
 ## 설치 및 실행
 

SECURITY.md

@@ -2,6 +2,11 @@
 
 ## Reporting a Vulnerability
 
-Please report a vulnerability to [email protected]
+Please report a vulnerability to `[email protected]`.
+
+## What not to report
+
+ * Firebase initialization tokens. The Firebase tokens are really public: they must be included into client applications and consequently are not private by design.
+ * Exposed `/pprof` or `/expvar`. We know they are exposed. It's intentional and harmless.
+ * Exposed Prometheus metrics `/metrics`. Like above, it's intentional and harmless.
 
-Please do not report Firebase initialization tokens. The Firebase tokens are really public: they must be included into client applications and consequently are not private by design.

chatbot/python/.gitignore

@@ -0,0 +1 @@
+.tn-cookie

chatbot/python/chatbot.py

@@ -301,14 +301,15 @@ def read_auth_cookie(cookie_file_name):
     return schema, secret
 
 def on_login(cookie_file_name, params):
+    global botUID
     client_post(subscribe('me'))
 
     """Save authentication token to file"""
     if params == None or cookie_file_name == None:
         return
 
     if 'user' in params:
-        botUID = params['user'].decode("ascii")
+        botUID = params['user'].decode("ascii")[1:-1]
 
     # Protobuf map 'params' is not a python object or dictionary. Convert it.
     nice = {'schema': 'token'}

docs/API.md

@@ -1,66 +1,67 @@
-<!-- TOC depthFrom:1 depthTo:6 withLinks:1 updateOnSave:1 orderedList:0 -->
+<!-- TOC depthfrom:1 depthto:6 withlinks:true updateonsave:true orderedlist:false -->
 
 - [Server API](#server-api)
-	- [How it Works?](#how-it-works)
-	- [General Considerations](#general-considerations)
-	- [Connecting to the Server](#connecting-to-the-server)
-		- [gRPC](#grpc)
-		- [WebSocket](#websocket)
-		- [Long Polling](#long-polling)
-		- [Out of Band Large Files](#out-of-band-large-files)
-		- [Running Behind a Reverse Proxy](#running-behind-a-reverse-proxy)
-	- [Users](#users)
-		- [Authentication](#authentication)
-			- [Creating an Account](#creating-an-account)
-			- [Logging in](#logging-in)
-			- [Changing Authentication Parameters](#changing-authentication-parameters)
-			- [Resetting a Password, i.e. "Forgot Password"](#resetting-a-password-ie-forgot-password)
-		- [Suspending a User](#suspending-a-user)
-		- [Credential Validation](#credential-validation)
-		- [Access Control](#access-control)
-	- [Topics](#topics)
-		- [`me` Topic](#me-topic)
-		- [`fnd` and Tags: Finding Users and Topics](#fnd-and-tags-finding-users-and-topics)
-			- [Query Language](#query-language)
-			- [Incremental Updates to Queries](#incremental-updates-to-queries)
-			- [Query Rewrite](#query-rewrite)
-			- [Possible Use Cases](#possible-use-cases)
-		- [Peer to Peer Topics](#peer-to-peer-topics)
-		- [Group Topics](#group-topics)
-		- [`sys` Topic](#sys-topic)
-	- [Using Server-Issued Message IDs](#using-server-issued-message-ids)
-	- [User Agent and Presence Notifications](#user-agent-and-presence-notifications)
-	- [Trusted, Public, and Private Fields](#trusted-public-and-private-fields)
-		- [Trusted](#trusted)
-		- [Public](#public)
-		- [Private](#private)
-	- [Format of Content](#format-of-content)
-	- [Out-of-Band Handling of Large Files](#out-of-band-handling-of-large-files)
-		- [Uploading](#uploading)
-		- [Downloading](#downloading)
-	- [Push Notifications](#push-notifications)
-		- [Tinode Push Gateway](#tinode-push-gateway)
-		- [Google FCM](#google-fcm)
-		- [Stdout](#stdout)
-	- [Video Calls](#video-calls)
-	- [Messages](#messages)
-		- [Client to Server Messages](#client-to-server-messages)
-			- [`{hi}`](#hi)
-			- [`{acc}`](#acc)
-			- [`{login}`](#login)
-			- [`{sub}`](#sub)
-			- [`{leave}`](#leave)
-			- [`{pub}`](#pub)
-			- [`{get}`](#get)
-			- [`{set}`](#set)
-			- [`{del}`](#del)
-			- [`{note}`](#note)
-		- [Server to Client Messages](#server-to-client-messages)
-			- [`{data}`](#data)
-			- [`{ctrl}`](#ctrl)
-			- [`{meta}`](#meta)
-			- [`{pres}`](#pres)
-			- [`{info}`](#info)
+  - [How it Works?](#how-it-works)
+  - [General Considerations](#general-considerations)
+  - [Connecting to the Server](#connecting-to-the-server)
+    - [gRPC](#grpc)
+    - [WebSocket](#websocket)
+    - [Long Polling](#long-polling)
+    - [Out of Band Large Files](#out-of-band-large-files)
+    - [Running Behind a Reverse Proxy](#running-behind-a-reverse-proxy)
+  - [Users](#users)
+    - [Authentication](#authentication)
+      - [Creating an Account](#creating-an-account)
+      - [Logging in](#logging-in)
+      - [Changing Authentication Parameters](#changing-authentication-parameters)
+      - [Resetting a Password, i.e. "Forgot Password"](#resetting-a-password-ie-forgot-password)
+    - [Suspending a User](#suspending-a-user)
+    - [Credential Validation](#credential-validation)
+    - [Access Control](#access-control)
+  - [Topics](#topics)
+    - [me Topic](#me-topic)
+    - [fnd and Tags: Finding Users and Topics](#fnd-and-tags-finding-users-and-topics)
+      - [Query Language](#query-language)
+      - [Incremental Updates to Queries](#incremental-updates-to-queries)
+      - [Query Rewrite](#query-rewrite)
+      - [Possible Use Cases](#possible-use-cases)
+    - [Peer to Peer Topics](#peer-to-peer-topics)
+    - [Group Topics](#group-topics)
+    - [sys Topic](#sys-topic)
+  - [Using Server-Issued Message IDs](#using-server-issued-message-ids)
+  - [User Agent and Presence Notifications](#user-agent-and-presence-notifications)
+  - [Trusted, Public, and Private Fields](#trusted-public-and-private-fields)
+    - [Trusted](#trusted)
+    - [Public](#public)
+    - [Private](#private)
+  - [Format of Content](#format-of-content)
+  - [Out-of-Band Handling of Large Files](#out-of-band-handling-of-large-files)
+    - [Uploading](#uploading)
+    - [Downloading](#downloading)
+  - [Push Notifications](#push-notifications)
+    - [Tinode Push Gateway](#tinode-push-gateway)
+    - [Google FCM](#google-fcm)
+    - [Stdout](#stdout)
+  - [Video Calls](#video-calls)
+  - [Link Previews](#link-previews)
+  - [Messages](#messages)
+    - [Client to Server Messages](#client-to-server-messages)
+      - [{hi}](#hi)
+      - [{acc}](#acc)
+      - [{login}](#login)
+      - [{sub}](#sub)
+      - [{leave}](#leave)
+      - [{pub}](#pub)
+      - [{get}](#get)
+      - [{set}](#set)
+      - [{del}](#del)
+      - [{note}](#note)
+    - [Server to Client Messages](#server-to-client-messages)
+      - [{data}](#data)
+      - [{ctrl}](#ctrl)
+      - [{meta}](#meta)
+      - [{pres}](#pres)
+      - [{info}](#info)
 
 <!-- /TOC -->
 
@@ -601,6 +602,21 @@ The `stdout` adapter is mostly useful for debugging and logging. It writes push
 
 [See separate document](call-establishment.md).
 
+## Link Previews
+
+Tinode provides an optional service which helps client applications generate link (URL) previews for inclusion into messages. The enpoint of this service (if enabled) is located at `/v0/urlpreview`. The service takes a single parameter `url`:
+
+```
+/v0/urlpreview?url=https%3A%2F%2Ftinode.co
+```
+The first several kilobytes of the document at the given URL is fetched by issuing an HTTP(S) GET request. If the returned document has content-type `text/html`, the HTML is parsed for page title, description, and image URL. The result is formatted as JSON and returned as
+
+```json
+{"title": "Page title", "description": "This is a page description", "image_url": "https://tinode.co/img/logo64x64.png"}
+```
+
+The link preview service requires authentication. It's exactly the same as authentication for [Out of Band Large Files](#out-of-band-handling-of-large-files).
+
 ## Messages
 
 A message is a logically associated set of data. Messages are passed as JSON-formatted UTF-8 text.

docs/drafty.md

@@ -192,7 +192,7 @@ _IMPORTANT Security Consideration_: the `val` and `ref` fields may contain malic
   { "tp":"EX", "data":{ "mime":"application/json", "val": { "seq": 3, "resp": { "confirmation": "some-value" } } } }
   ```
   * `url`: issue an `HTTP GET` request to the URL from the `data.ref` field. The following query parameters are appended to the URL: `<name>=<val>`, `uid=<current-user-ID>`, `topic=<topic name>`, `seq=<message sequence ID>`.
-  * `note`: send a `{note}` message to the current topic with `what` set to `data`.
+  * `note`: send a `{note}` message to the current topic with `what` set to `data` (not currently implemented, contact us if you need it).
   ```js
   { "what": "data", "data": { "mime": "application/json", "val": { "seq": 3, "resp": { "confirmation": "some-value" } } }
   ```

docs/faq.md

@@ -98,3 +98,12 @@ The test database has a stock user `xena` which has root access.
 
 ### Q: What is the proper way to format gRPC {pub content}?<br/>
 **A**: The gPRC sends `content` field of a `{pub}` message as a byte array while the client applications expect it to be valid JSON. Consequently, you have to format the field to be valid JSON before passing it to gRPC. For example, to send a plain text `Hello world` message you have to send a quoted string `"Hello world"`. In most cases the string you pass to the gRPC call would look like `"\"Hello world\""` or `'"Hello world"'`.
+
+
+### Q: How to fix PostgreSQL initialization failing with 'missing database' error?<br/>
+**A**: PostgreSQL has a (mis)feature: a DB connection must always select a database. If the connection tries to use a database (even with intent to create it) which does not exist, the connection fails. When Tinode is started for the first time, it tries to create a database, usually `tinode` (see `tinode.conf`, `"store_config": {"adapters": {"postgres": {"DBName": "tinode"}}}`. The database `tinode` obviously does not exist, so Tinode connection falls back to 'default' database which has the same name as the name of the connecting PostgreSQL user. The default configuration specifies user as `postgres` (`"User": "postgres"`), the database `postgres` always exists, so the connection succeeds and everything works as expected. But if you change the user to anything other than `postgres`, let's say `tinodeadmin`, then trouble starts: the database with the name `tinodeadmin` does not exist and the connection fails. If you want to change the user name to anything other than `postgres`, then you must create either a database `tinode` (or whatever you named your Tinode database) or an empty database with the same name as your user `tinodeadmin`. For example:
+```
+$ psql
+	postgres=# create database tinode;
+	exit
+```