Jump to content


KT Walrus

  • Content Count

  • Joined

  • Last visited


About KT Walrus

  • Rank
    Community Regular
  • Birthday 11/11/2014

Recent Profile Visitors

The recent visitors block is disabled and is not being shown to other users.

  1. Read/write separation isn't the same as read-only access to the database from user HTTP requests. Separating the writes asynchronously wouldn't help much unless you separate out the whole transaction (e.g., run _createMember()) asynchronously in an admin task on behalf of the currently logged-in member. There is still a possibility for a hacker to find a hole to execute an update query they craft, but it does make it more unlikely (since only the admin task has permission to update tables in the database). You could extend this further to have multiple admin mysql users so the specific mysql user that only has update permissions on the tables the transaction is supposed to execute. This would provide even greater security as most transactions would not allow promoting a member to the admin group (which is the holy grail for hackers, I'd think). So, @bfarber, I argue that the benefits of this proposal to make IPS5 transaction safe and asynchronous (out of the HTTP request processing) out-weigh the small re-factoring of PHP code required.
  2. Consider re-organizing PHP code that updates the database so that these updates can be performed asynchronously and not by HTTP while the user waits for the page to load. For example, the _createMember() function called during registration could be executed asynchronously (including sending the verification email). Some existing code may have to change to make all database writes done asynchronously (for example, showing the post immediately after queuing the database updates might not have been committed yet to the database, so the UX might have to change slightly to show a "post is being moderated" and should be approved in a few minutes). This reorganization of database updates has the following advantages: These PHP functions can be executed by a system task and doesn't slow down the page loads waiting on all the updates to complete. If all database writes are off-loaded to a system task (except maybe those in the ACP), the mysql database user needs only read-only permissions to the database (making IPS5 even less vulnerable to unauthorized database writes and read-only MySQL slaves may be utilized to add reliability and application performance). The admin could configure their installation to use RabbitMQ (or even Redis) to queue these system tasks for execution by one or more PHP workers. Each individual task could be marked as "may be executed in parallel" or "must be executed serially". Parallel execution probably wouldn't be necessary except for the busiest sites. The PHP workers would normally execute these tasks within a few seconds of being queued (especially if multiple PHP workers configured). The entire set of database writes can be applied to the database in a single SQL transaction (using BEGiN/COMMIT statements surrounding the call to the PHP function) instead of using AUTOCOMMIT for each individual query. Using SQL transactions properly is better for database integrity and for database replication (used by sites that require High Availability). Many other minor advantages not detailed above. Using asynchronous tasks to perform all database updates also would permit doing other functions asynchronously (like sending notifications, queuing outgoing emails, even possibly invalidating a Redis cache only when data in the cache needs to be updated due to database changes to particular rows in a table). More and more features in the suite might be done asynchronously over time (like video and audio transcoding, etc).
  3. It is super easy to embed mp3 files by using HTML5 <audio> tag. Don't know why IPS hasn't done this already. The suite embeds video using the HTML5 <video> tag though. Note that you need an HTML5 Audio/Video player to provide adaptive streaming of video (HLS or DASH) which is really a requirement for any site with lots of video.
  4. Don't forget about Desktop PWA support as well. Really want my users to only use the PWA app in the future after downloading the app from their first visit to my site. And, the UI should make the experience as good as the underlying platform support can make it. I want my users to think my site is an app and not just a bookmark in their browser. Bookmarks can quickly get forgotten and it isn't that great for long term engagement. An app with SMS notifications is hard to forget about unless they explicitly delete the app or block the notifications.
  5. I'm asking that version 5 have as much support for both desktop and mobile PWAs. The lack of native apps for my website is a real impediment to user growth and PWA technology will probably be fully supported on all platforms by the time version 5 ships. I hope this is a very high priority at IPS for version 5. Here are some Google links about PWAs: Overview Desktop PWAs Progressive Web App Checklist
  6. I don't know much about Vue.js but when I watched a few YouTube videos to learn more, it seemed like it might fit well into the templating system and make junior coders like me more productive mixing HTML, CSS, and Javascript in the same page. And, IPS5 seems like the only opportunity to make such a big change like this. I really want IPS5 code to be pushed to the edge (either on the user's device or supporting Cloudflare Workers). I can see that there is a chance that PWA apps will eventually be as nice as native apps on mobile devices within a year or two (if Apple finally gets onboard) and you need to write most of the user's interactions with your website in JS (and cache data locally) to do it. For the site that I'm working on, it would be good to be able to cache the latest 1000 topics (for index lists), a latest threads in the unread activity stream and the mini-profiles of the most recent 1000 members locally within the browser and use that cache for building various lists/pages just like an email client syncs messages/indexes locally when it periodically syncs with an IMAP server. This makes email apps very responsive while IPS4 is not as good a UX on mobile. Reading your mail is instantaneous, but browsing forums is not (especially when not on Wi-Fi where network latency is usually very high).
  7. Please consider using Vue.js in 5.0. This framework would integrate well into the theme templates to move more processing into the browser and is very user-friendly to work with.
  8. Turns out that the company behind tus.io are implementing a very nice looking photo and video uploader called Uppy at https://uppy.io. This open source project appears to be in very active development, but looks very interesting and maybe something to replace Plupload: After playing around with this, I think it will be great for uploading photos and videos on my site since it supports uploading from Google Drive, Dropbox, Webcam, and other places users might have their videos and not just on their devices.
  9. I am working on using Cloudflare Stream for user videos on my site. Cloudflare Stream uses tus to upload the large video files to Cloudflare. I searched the web for a tus client implemented in PHP so I could add support for uploading video files to Cloudflare to my ICS4 apps. I found ankitpokhrel/tus-php GitHub repos that implements both a PHP tus server and a PHP tus client. Researching this further led me to tus.io where you can find out everything about this protocol including an official implementation of a Javascript tus client at the tus/tus-js-client GitHub repo. Cloudflare and Vimeo apparently use tus. Seems to me that IPS should consider adding a tus file uploader/client/server to ICS4. Tus could possibly replace the existing Plupload JS, or maybe add a large file video/photo upload page available from a link in the user menu labeled "Upload Media...". This would give the user the ability to upload very large files similar to Plupload's Queue Widget using tus which apparently has full support for things like checksums, pause, and resume on a page that would allow the user to upload multiple files in batch (where the user identifies the files to upload upfront, starts the uploading process with the ability to suspend/resume/cancel if needed). Hopefully, IPS devs can look into adding tus for file uploads in a future version of the suite and possibly deprecating Plupload or making it an admin choice as to which JS to use for file uploading by users.
  10. I support this idea. Also, on mobile, it would be nice to increase the default font-size for pages. This would help a bunch for those with less than 20/20 vision, myself included. I hate having to zoom the page to read text comfortably since the entire width of the page no longer shows in the viewport (leading to a bunch of swiping left and right). Perhaps a Switch to Dark mode and an increase default font-size could be added to the hamburger menu to quickly control these theme settings by the user.
  11. I was disappointed to see that 4.4 didn't "fix" embedding videos in email notifications. I figure that the current approach to just embedding text saying to view the video in a browser was just a stop gap measure until you could support it properly. That is, the preview image should be embedded in the email linking the image to the website video (for actual streaming of the video). Here is a pic of what it looks like now in email: Here is a pic of what it looks like now in a browser: What it looks like in email should be the same as what it looks like in a browser. Of course, not all email clients support actually streaming the video in the client, but at least it should look visually the same with clicking on the play button taking you to your browser to play the video in the browser.
  12. The biggest improvement would be that after uploading a big file, you wouldn't try to download a thumbnail to show in the editor. I tried uploading a 200MB video file with the current uploader from my iPhone and it wasn't a good experience. This was with 4.3.6 (not the 4.4 beta). Also, a big win for uploading larger files is the user controls when the uploads start. This Queue Widget is a much better UX, IMHO. They can upload 4 or 5 videos at a time by queuing them within the uploader and press "Start upload" when they want the uploads to begin. Same goes for multiple photos (uploaded from my iPhone - photos are pretty large files if you allow uploading the original images). Much better I think to queue the uploads and start them all at once (especially on 3G/4G networks). I'm adding support for Cloudflare Video Streams on my site. This will require the users to upload their video files first. I plan to then run Zencoder over the uploaded files to transcode them before uploading to Cloudflare for streaming to the viewers. This requires that I provide an efficient uploader (which I've already prototyped using the Plupload Queue Widget and this seems to work well - at least from my iPhone/Mac). BTW, I see that Plupload has the option to resize photos client-side and rename files before they are uploaded (I think). The Queue Widget could give the user the ability to change the default settings before uploading (or, if ICS4 doesn't already support it, to add ACP settings for resizing/renaming client-side).
  13. The Plupload Queue Widget is a perfect interface for uploading large files like videos. Please consider implementing a new Form Field Helper for uploading Media (photos & videos) using the Plupload Queue Widget. It should be very simple to code since the suite already uses Plupload to upload attachments. Please make the helper as mobile friendly as possible as users are probably going to upload videos directly from their mobile devices. Also, add an "Edit Filenames" button to the "Add Files" and "Start Upload" action buttons. This would allow the user to set a more friendly filename for the uploaded video or photo.
  14. I have a few forums that I keep some pinned topics in to help those new to the forum. For those forums that have multiple pinned topics, I want to set the sort order (still at the top of the list) instead of ordering by date. Could you add an option to moderate the order of pinned topics within a forum in ICS 4.4?
  15. I'm in the process of adding support in my apps for the RabbitMQ message broker to separate admin tasks (that may be performed asynchronously) from the app's front-end. RabbitMQ implements the AMQP 0-9-1 messaging protocol (with other protocols supported via rabbitmq-plugin). This is turning out to be quite nice. Just like ICS4 added support for Redis database cacheing, I would like to see the next version of ICS4 add support for AMQP 0-9-1 asynchronous message passing to backend admin tasks. For example, it would be better if handling emails in and out of ICS4 where done through RabbitMQ to start with. The parameters for an outgoing email message could be queued to one or more admin task clients that wait on new email parameters for outgoing messages and processes the message template with the parameters to form a complete mail message and then sends them at out using an SMTP service. Note the admin workers can be written in PHP and run within the ICS4 environment (just like current ICS4 tasks are run from a cronjob). They simply have to wait on messages from a queue in RabbitMQ and process these messages (in this case, sending or receiving emails). For incoming emails, it is rather easy to configure Postfix to pipe incoming emails to a PHP file for queuing in RabbitMQ. A separate admin worker would watch this queue and ingest any properly addressed messages and store those messages (along with attachments) in the database. This could be implemented for support messages in Commerce, but could also be used to implement "post by email" functionality in the suite (like to upload photos to Gallery via email). For my apps, I want all database updates to be done asynchronously via RabbitMQ. This means that I can't use auto-increment primary keys but must use global UUIDs for my database tables. That is, instead of updating the database with a series of INSERT/UPDATE queries directly, my app will generate a UUID for the post and queue all data necessary to update the database through RabbitMQ to an admin worker that knows how to turn this data into an application content post and update the database asynchronously. I plan on cacheing the post data in Redis before queuing to the backend so that my app can show the new post to the user even before it makes it into the database (since RabbitMQ is asynchronous message delivery). There are many other features in ICS4 that could be offloaded to RabbitMQ over time to make ICS4 even faster and more scaleable (and highly available). My goal is to have my ICS4 frontend only have read access to the MySQL database and use RabbitMQ and Redis as much as possible. This would allow better security to my database since the MySQL user used by ICS4 frontend would be restricted from performing update queries while the actual updates would be done by an admin worker using an admin MySQL user/password. BTW, it appears to be very easy to run a RabbitMQ Cluster in High Availability mode using containers. Such a cluster would survive temporary or long term unavailability of a number of nodes in the cluster without losing any queued messages. Workers could be scaled up/down as needed rather easily since their job is to asynchronously process front-end requests. You can even implement a queue with only one worker processing all messages in the queue. This allows you to "serialize" the database requests (which might come in handy for some tasks - I have one such task in my ICS4 apps that really benefits from being applied serially). The backend workers can apply multiple update queries in a single MySQL transaction to make it harder for the database to become inconsistent. Finally, failures in a backend admin worker due to bad input or a permission issue can be reported to the user via ICS4 notification messages. That is, failure to apply all updates on new content creation could send a notification to the poster that the backend failed to add the new content to the database (because the database was not available for update) and that the admin worker had requeued the new content for retrying later. Just like an SMTP server may inform the sender that the message couldn't be delivered and will keep trying for a few days, the admin worker could send similar notifications to the poster (and maybe an SMS message to the ICS4 Admin) that there was a problem in the backend.
  • Create New...