BN_IT_Support

Hi Joe, What are you asking for? Are you asking for someone to quote you for doing the work? (Which I personally don't have the time to do.) Are you asking whether it would be technically possible to develop an app something like that? (For which the answer is "yes" - it would be technically possible!) John

It may well not be relevant - but if the hosting company enabled SELinux then that could be the problem. I know that whenever I have had SELinux enabled then trying to get Invision to do anything is a nightmare! John

Hi - you need to get the type of relationship straight and clear. If you only want a 1-to-many (or many-to-1) relationship then you can probably do it with Pages Databases. If you actually want/need a many-to-many relationship (which is what I think you will need) then it will almost certainly be easier to write an application to do it. I won't say that it would be "impossible" to do many-to-many with Pages Databases but I believe that it would be really hard work. You are talking about just two tables/databases but to do many-to-many you need an additional table/database that sits between the other two and links the records together. You stated the problem as having: Many matches Many players Requiring a relationship so that you could add many players to a match As stated it appears to be a 1-to-many (that is each match can have many players) and that could be done with Pages Databases relatively easily. But, appearances can be deceptive. The above (1-to-many) will only work if each player can only play in 1 match (as that's where the "1" comes form in 1-to-many). It is far more likely that your players can play in many matches -- i.e. they play in one match this week-end and a different match next week-end and so on. So, you have both "matches can have many players" and "players can play in many matches" and that is a many-to-many relationship that becomes extremely complex to implement with Pages Databases. John

Marc, Thanks for the clarification/correction. John

I'm not 100% confident in this, so someone may want to correct me 😉 I believe that if you turn this off then moderators can still moderate content in clubs that they can access. In other words, 'access' becomes the normal user type of access for the type of club, but once the moderator can access the club then they can moderate the content. Specifically, if you have a private (invisible) club or a club that you must join in order to see content then that content would be invisible to a moderator unless they also join the club. In order to make this work you would have to ensure that all moderators were invited to all clubs and that the moderators actually joined the clubs. You would have to take sanctions against any club leaders who threw the moderators out. Alternatively, you would need to create a plugin or application to scan all moderators daily and join them back into clubs if they get thrown out (or if they have never joined). When you turn the switch on then what it does is gives moderators access to private/hidden clubs even if the moderators are not members. John

Hi, You said 'database' - did you mean a 'Pages Database'? If so, that is really only a single table. Your question really relates to standard SQL relationships - are they one-to-many, many-to-one or many-to-many. If you have one-to-many/many-to-one relationships then it is fairly easy - for example, teams and players. You have a team with a team_id. You have many player records and every player record has a reference to the team_id. In that way you can list all the teams and for each team you can find the players that have a matching team_id so you can find the members of the team (or, at least, the squad). Your scenario seems to invite many-to-many relationships so you will need 'intermediate' records that 'cross-reference' between other records. Personally, I think that gets overly complex if you try to do that within Pages Database and I would write my own Application (within Invision) and define multiple tables. (We did this for our 'volunteer recruitment' application/database - we had many examples of many-to-many, for example users could create many applications (for different roles) and we could have multiple applications for a single role.) In your case you have: team: team_id, chairman, coach, (address if you only cater for home and away, but if you include neutral ground - e.g. Wembly for the Cup Final then you would need 'venue' records and include the 'venue_id' instead of the address). players: player_id, name, DoB, default_position (might not always play every match in the same position), team_id (assuming they can only belong to one team then it is a many-to-one relationship, but if you want to track players over time as they switch teams then you need an intermediate many-to-many relationship) venues (not needed if you only cater for home and away and include the address in the home team record): venue_id, address match: match_id, date, venue_id team-to-match (many-to-many cross reference between teams and matches): xref_id, match_id, team_id, score (by this team as the other team has its own score). You typically have two of these team-to_match records linked to every match record... player-to-match (many-to-many cross reference between players and matches): xref_id, player_id (this player was actually picked for this match), position (this player played this position in this match), score (this player scored this much in this match) Then you need some fancy SQL 'joins' to pull the information that you want... John

Hi, I don't fully understand what you're doing, so this might entirely miss the point 😉 I have never done exactly what you're doing, but we have done something simpler - doing lookup of ZIP to lat/long using various API providers. I think you're going to find that all APIs are different - either because they took short cuts and implemented whatever seemed easiest on the day or else for competitive advantage. Either way, you will find you need to code to multiple different APIs. I would design this with a separate file (possibly 'Source') for each API. Each file would implement an interface for your core - say 'get' and 'search' - and would then implement the API specific stuff such as the actual network/HTTP calls and then reformatting the response to the standard structure used in your core. In this model, Zapier would be just one more file following the same general structure - nothing special. If you have a significant number of API services (files) then the question arises of how to you call them. Do you just hard code it and go round a loop calling each of your 5 to 10 API services in sequence or do you implement a 'discovery process' - similar to what Invision does to discover your extensions so it knows to call them. Obviously, a discovery process is slightly more compute intensive but it can make it easier to manage as you just put your files in a specific folder and they get discovered and called. How do you schedule your code to run? If calling only one service then you could do it as and when the user creates a record as the additional processing time and delay is unlikely to give a bad user experience. But if you have 5 to 10 services then you almost certainly need to queue your processing into the background so the user interface is not delayed. John

Hi, I agree with teraByte - write a custom application to do this - you possibly only need one significant script file. I would guess it is less than 20 lines of script: define 'where clause' ($where) that compares the join date against your date range create iterator of selected Members: new \IPS\Patterns\ActiveRecordIterator( \IPS\Db::i()->select( '*', 'core_members', $where), 'IPS\Member' ); Loop through members calling ->flagAsSpammer() At end of loop you're all done. John

Did you save your login credentials in the browser this time? (I'm not sure how that would play with your 'logout' - except that typically 'logout' is on the server and affects your cookies whereas credentials saved in the browser are - well - credentials saved in the browser.) John

Hi, I'm largely guessing what you mean - thus possibly guessing incorrectly. It would make it clearer if you provided more detailed examples of what you want. My guess: In PHP something like this might work: $url = \IPS\Http\Url::external("https://{$service}:{$port}/rest-of-url"); or in a template file (PHTML): {{$url = \IPS\Http\Url::external("https://{$service}:{$port}/rest-of-url");}} Then you can use the $url variable in your text/html - either in a template file or else text embedded in your PHP: <a href="{$url}">Click this</a> John

Hello. I started with your distance calculation in Item.php: $selectClause .= ', ( 3959 * acos( cos( radians(' . $location['lat'] . ') ) * cos( radians( ' . static::$databasePrefix . 'latitude' . ' ) ) * cos( radians( ' . static::$databasePrefix . 'longitude' . ' ) - radians( ' . $location['lon'] . ' ) ) + sin( radians( ' . $location['lat'] . ' ) ) * sin( radians( ' . static::$databasePrefix . 'latitude' . ') ) ) ) AS distance'; I converted this to a PHP calculation (and I believe that I got the conversion correct!) $distance = (3959 * acos(cos(deg2rad($lat)) * cos(deg2rad($long)) * cos(deg2rad($event_long) - deg2rad($long)) + sin(deg2rad($lat)) * sin(deg2rad($event_lat)))); The problem is that this version of the calculation gave me 'NAN' (Not a Number) for the values that I tested. After a quick Google for the algorithm I found the 'Haversine formula' and that seems to work OK. $distance = 3959 * acos(sin(deg2rad($lat))*sin(deg2rad($event_lat))+cos(deg2rad($lat))*cos(deg2rad($event_lat))*cos(deg2rad($event_long)-deg2rad($long))); I conclude that your calculation in Item.php is incorrect. John

Hi, I don't have the answer, but... I find it interesting that you would use your own editor on a Theme hook. I would agree that using your own editor on a Code hook is far more convenient than using adminCP editor, but for a Theme hook ??? Personally, I find the adminCP editor more convenient as it sorts out all the escapes (among other things). What motivates you to use your own editor? Secondly, if this used to work before 4.7.16 and if you have NOT made any changes to your editor (either configuration or version update) then I can understand your point of view - that it should not have been broken. That said, the Invision comments say "do not remove" and that implies "do not modify, either" and your editor has clearly modified the lines - what was '/*comment*/' is changed by your editor to '//comment' which is not the same thing. John

Hello, I believe that this is a more serious consequence of removing the 'Submit Details' button in 4.7.15. My previous topic about the 'Submit Details' button was because one of our plugins was triggering off the click on the button. Fair enough - that one is our problem and we have to find something else to trigger off. This is a different and more serious issue. We have a number of custom applications and plugins so I disabled all of them on the test system and re-tested. The problem as described below is with all our custom or 3rd party apps and plugins disabled. When users (members) submit images to the gallery we require them to provide both a title and a description for the image. That works fine on a laptop as details are automatically saved as you move from image to image and there is no longer any need for the 'Save Details' button. The problem occurs on phones and other narrow screen devices. (The problem can be reproduced on a laptop by setting the window narrow like on a phone.) For example, on a phone add 4 images. Click the first image to add the details => the details popup covers the other images so it is impossible to select them => the details popup hides the submit button - although it is possible to rotate the screen/device to landscape mode and the submit button peeps out from behind the details popup. In short, if you need to add titles, descriptions, etc. to several images then the images must be added and uploaded one at a time. Adding/uploading multiple images at a time results in inability to set details for more than one of them. This is not a problem with the phone. It has been tested and is the same on iPhone, Android and laptop (when window is set narrow). Regards, John

Jim, Thanks for your quick reply. My apologies - I thought that BNTEST was running with the 4.7.15 dev files -- I had upgraded BNTEST on 24th Jan and had downloaded the matching dev files on the same day. Evidently, I did not apply the dev files so the dev files must have been 4.7.14 which is why I did not spot the problem. No - we'll not be adding the missing lines to our version of the theme. We have a strict policy of not changing any standard files because changed files quickly become a nightmare to maintain. If the button is not needed in the standard software then we have no reason to add it back in. (Except we were forcing users to enter unique titles and descriptions by triggering off the 'Save Details' button so we'll have to find something else to trigger off 😉) Thanks John

Hello Support, We upgraded the BNTEST site pretty much as soon as upgrades become available so it has been upgraded in stages including 4.7.12, 4.7.13, 4.7.14 an 4.7.15. We did not notice any problems because we run with IN_DEV all the time and the wrapper.phtml file (used in IN_DEV) still has the button! We upgraded the BN site (our production site) on Sunday morning directly from 4.7.12 to 4.7.15 - being production we do not use IN_DEV and as the upgrade was performed in a single step we could not say which of versions 4.7.13, 4.7.14 or 4.7.15 introduced the bug - all that we can say with certainty is that both BN and BNTEST show the same bug (if INDEV is disabled) and that the upgrade path of BN and BNTEST was different and done at different times. In simple terms, the standard Invision Theme gallery->front->submit->wrapper has 6 lines missing from close to the end. There are a number of <li> elements on the details form and then a {{foreach loop}} and then the button should exist on the form. Without the button it is possible to enter details for each image (including title and description) but it is not possible to save the details. The missing lines are: <li class='ipsFieldRow ipsClearfix'> <button type='button' class='ipsButton ipsButton_light ipsButton_fullWidth ipsButton_medium' data-role='saveDetails'>{lang="submit_go_back"}</button> <p class='ipsType_light ipsType_medium ipsType_right ipsSpacer_top ipsSpacer_half ipsHide' data-role='savedMessage'> <i class='fa fa-check'></i> {lang="image_info_saved"} </p> </li> I'd like to have this fixed ASAP 😉 I'd be quite happy to insert the lines into the database myself but I probably ought to follow some official procedure and if you have a patch for the issue then I should use that. Thanks very much. John

As a general opinion on "Pages DB"... They are not really "databases" in SQL terms -- but SQL "tables" in the Invision database. When you are considering which sort of SQL table to use (within Invision) you have two choices: "Pages DB" (as you have mentioned) SQL tables as defined under the 'Database Schema' tab for an Application Pages DB works well for: Where you only need a single table for all your data - for example, in the classic database design if you only need a "products" table then it will work well. If you actually need a "products" table and a "suppliers" table with cross references then it does not work so well - and in any case you would need two Pages DB (one for products and one for suppliers) Where the solution involves a human adding and editing records it works well - for example, where you have a human adding products (descriptions, part numbers, prices, etc.) to a product table. We probably use between 5 and 10 Pages DB for various unrelated things such as "news articles", "places to go" and so on. Using Pages DB automatically gives you various widgets such as "Database filters" which is good. I would strongly recommend that you use the standard display templates wherever possible. If you create your own display templates based on the standard display templates then that effectively bases your template on a snapshot of the standard template - you don't benefit from future improvements and bug fixes to the standard. We have some of our own templates from 6 or 8 years ago and regret not putting more effort into display formatting for individual fields. With some clever display formats you can combine fields to produce complex output (for example, latitude and longitude fields to display a map) and if you need to do conditional display of some fields dependent upon the contents of others then you can add a dummy field (with a non-blank default value so it triggers display) and you can write custom code to display the contents of several other fields according to the rules that you want to apply. Do the fancy stuff in a field display rather than a template wherever possible. When you write a scheduled task (for example) to update the data in a Pages DB there are a couple of (minor) extra steps that you need to take in order to write the data. Firstly, you need to know the classname that will be used to write to the correct table and this will be something like \IPS\cms|\Records23 where 23 is the database number. So, you have to look up the database number from your database key (I recommend against hard coding the 23 😉 ). Secondly, if you have fields for name, address, region, country, phone, etc. then the names in the table will be something like field_91, field_92, field_93, field_94, etc so you have to 'map' from the field key to the real field name in the database. Not difficult, but that's what you have to do. Pages DB will work OK where you have a scheduled task to update the data once per day (for example - it could obviously be much more frequently if you wanted). A scheduled task will work fine where nearly all the data items/records are accessed every day - but what about a scenario where only 5% of records are accessed each day? Doing a daily scheduled update of all records means that 95% of what you retrieve is not going to be used in the next 24 hours. I rather liked Nathan's suggestion of retrieving data on demand (i.e. every record has a lifetime field that you add so if you want the record but the lifetime has expired then you retrieve the data through the API). That would mean that you would not waste resources retrieving data that is not being used. Unfortunately, "retrieval on demand" would be extremely difficult or impossible to implement using Pages DB - unless you write your own very fancy templates and in that case you could do better to write an Application to do the entire job. Updating Pages DB from a scheduled job would work easily. Application tables will work well for: You need several tables with relationships between them (e.g. products and suppliers) You need an application that does a lot of processing of records (rather than simple human entry of records as indicated previously) Access to records is more intuitive - field names are what you want to call them rather than being 'mapped'. Also, access to the correct table is very simple as you need to create a sub-class of \IPS\Patterns\ActiveRecord that will get you to the correct table plus a whole load more stuff. If you have hierarchical data (for example, categories and records) then you should consider using the Node/Model/Item/Content model although that gets a lot more complicated it does give you access to many of the features that you see time and again in Invision. You have complete control so retrieving data on demand is much more simple than it would be using Pages DB. Regards, John

Would I be correct in deducing that you're referring to this (on a custom block)? If so, no, it would be the data that would be cached so you would lose control of how and when it would be refreshed. Certainly, you could use a custom block, but just make sure that you leave 'Cache this block' set to 'No' and then do your own caching. The way that Nathan Explosion suggests is by far the best way... Firstly, there is "how/when you retrieve the data": Scheduled polling would collect all the items even if some have not been updated. Daily polling would work OK if all items are updated every day but if only half the items are updated each day (for example) then you would be retrieving twice as many items as required. If you need the most up to date data for each item then you really need webhooks to notify you of changes - in the absence of webhooks you either set a very short scheduling interval or else use Nathan's solution with a short cache lifetime. You have now told us that you don't mind data which is not completely up to date (say - one day out of date) so webhooks are not essential. Nathan's solution where you load the cache on demand (i.e. only retrieve an item is it is not in your cache OR it is in the cache and already expired) - set the cache timeout to 12 hours or 24 hours or whatever depending how far out of date is OK. If the rate limiting still causes problems then you might want to (need to) randomise the cache lifetime a bit. For example, with a fixed cache lifetime of 1 day if you start with an empty cache and if the users demand all data in a very short time then you might exceed the rate limit. (Not likely, but possible?) With a fixed cache timeout the same thing will happen the next day as all items will expire at the same time. Randomise the cache time with +-60 minutes and lookups will be distributed over time very quickly. Secondly, there is "how/where to cache the data": If you only need a single lookup key per item (e.g. a name OR an id) then Nathan's suggestion of using the data store is the best way to go. (I've used it where I only need one key per item and it works very well.) If you need multiple lookup keys per item (e.g. name AND id so you sometimes retrieve from cache by name and sometimes by id) then the data store does not have that flexibility so you would probably need an SQL table with multiple keys - unless you double your storage usage by saving each item under both the name key and the id key. (I have multiple tables with multiple keys which is why I think that way 😉 ) John

If you're going to go down this path then I would be inclined to use a 'task' within your Invision Application - that way you can use the API to pull data directly into your application rather than pull the data with Zapier or a separate cron script and then post it into your application. An important question - how far out of date can you allow the data to be? If the data must be up to date within 5 minutes then your script will have to run every 5 minutes and if your application does not display all the data frequently then you may well generate more activity pulling data on a schedule than you would pulling it on demand. Is there a way (in the API) to detect data that has been updated - so that you don't have to pull all the data periodically? The most useful thing would probably be if the API includes webhooks that will fire and notify your application if a data item/record is created and when one is updated -- i.e. you then use your webhook to pull data that has changed and don't bother to pull data that has not changed. If there are no webhooks in the API as an alternative will the API let you 'list all records that have been updated in last 10 minutes' (for example) - that way you would use the list of recent updates to select which records to pull (i.e. only those that have been updated since last pull). Finally, I would be inclined to store the data in your own database (i.e. table defined by your application). Effectively, the database table acts as your cache and if your cache is only updated on data change (webhook or whatever) then it would probably be far more efficient than trying to use the Invision cache (which will have its own timeouts that are not necessarily in sync with updates to the data...) John

Hello, Version 4.7.15 compileFullEmail attempts to fold long headers by using the wordwrap function - but this does not work correctly as whitespace gets 'swallowed' and the extension line does not start with white space. The rule for Long Headers (those that exceed 998 bytes) is that they should be split so that the next part starts with white space -- i.e. crlf ma#y be inserted before the whitespace and NOT as a replacement for the whitespace. "unfolding" only works correctly if the follow on lines start with white space -- i.e. "Unfolding is accomplished by simply removing any CRLF that is immediately followed by WSP." Our particular failure scenario was when using our application to send to approximately 100 recipients as the $to field then exceeded 1,000 bytes and the To: header was split incorrectly. This then messed up the mail server processing resulting in failures to deliver. Quite probably different mail servers mess up in different ways - in our particular case the mail server put the extension line in the Subject: field and decided that there was no From: header. Regards, John

Hi, I have spent a number of days reading your IC5 development blogs and reviewing our plugins and applications to see what problems may arise and how we might solve them. Some changes are easy under your new system but others appear to have no obvious solution - and that will be a major problem for us. adminCP reports that we have 62 hooks in our applications and plugins - that is "hook files" and as some of our hook files actually contain a number of hooks (e.g. one of our Theme hook files actually hooks into 13 different places) we could have around 150 actual functional hooks. Many of the problems that we will face with IC5 appear to be because we cannot use hooks to remove functionality from the system. The following is a summary of some of the issues: Need to remove author (and any other personal information) from content displayed to certain groups. Our website is for a member organisation but also part of the website is visible to Guests (and a few other non-member groups) - and for security reasons we must conceal all personal information from Guests and non-member groups but allow it to be seen by members. We have 11 code hook files and 2 theme hook files to hide author information (replacing it with Guest information) as well as hiding the jsonLD information. Besides generic 'hide the author' it also took application specific hooks to hide download data, CMS data and calendar data - which also included RSVP sections of the calendar (no point in showing the RSVP if all the details are hidden). Hide "follow" buttons for Guests and all non-member groups. Hide comments and reviews from Guests and non-member groups. It is a member organisation so members can read content and post comments and reviews that can be seen by other members. Although we have content that can be viewed by Guests and other non-members we do not want the non-members to be able to see comments or reviews posted by members - there are two obvious reasons: (1) that some members post derogatory comments that are tolerated for viewing by other members but are not appropriate to show to non-members (2) part of the added value of being a member is that comments and reviews are visible as well as being able to post one's own comments and reviews. Disallow upload of attachments in various places - in particular, cannot add images to album descriptions, image descriptions, image comments, blog comments etc... Disallowing attachments is selective, so for example attachments can be added to forum topics and posts. Enforcement of "gallery rules". Simplistically, the gallery rule is that all uploaded images must have a meaningful and unique title and a similarly unique and meaningful description. Currently, this is enforced when uploaded images are submitted - the hook checks that suitable titles and descriptions have been entered and reports an error if the checks fail. Resizing (shrinking) images on upload. The standard IPS image size limits don't really work the way we need. File size (kB or MB) does not really relate to resolution (H and W in pixels) and in any case setting a limit gives a bad user experience as they don't know how to reduce image size to meet whatever limit we set. Our solution is to accept pretty much any image size and then reduce it to the size (HxW) that we want. This is selective according to where the image is to be stored so different resolutions are set for gallery, blogs, forum, profile, status updates, etc. Displaying maps - two actions: (1) filtering who can see maps (only member groups can) and (2) caching maps. The issue is that Google Maps started charging for use of the maps API - there is a free allowance every month but if we didn't take the above actions then we would massively exceed the monthly allowances and it would cost us a lot. Firstly, it is a member organisation so allowing non-members to run up costs (by viewing calendar maps) would not be acceptable so non-members cannot view maps - instead they are presented with a standard outline and a message stating that members can view the maps. Secondly, each event tends to be viewed multiple times by different members and if we did not take action then every view would generate a Google Maps API call which would cost us. We cache map lookups using the coordinates as the key and this reduces our costs. (This applies to other maps such as images and CMS/database maps as well.) Remove all references to the IPS newsletters and bulk mailing as we have an alternative system. (Our newsletters are managed through our membership system in conjunction with Mailchimp.) We have hooks to remove the standard newsletter widget and replace it with our own and another hook to remove notifications relating to newsletters/bulkmail. Add additional information to members displayed in the staff directory. Specifically, the hook adds the member's organisation email address (rather than their personal email address). Selectively hide poll results from SOME members for SOME polls. Specifically, forum polls are used for voting on AGM motions and it is believed that showing the current voting levels could influence voters so the results are hidden from all members (except the people collating the votes). Other polls (i.e. not AGM voting) behave normally. There are some that I have missed out that also don't have obvious solutions. Obviously, I've missed out all the hooks for which I believe there may be a solution (based on your blogs). How can I solve the above? Thanks. John

/usr/bin/curl https://remoteservices.invisionpower.com (from my server -- no browser available) returns immediately with no error. Using Firefox (from my laptop) returns <html><head></head><body></body></html>

Ditto - having the same symptoms right now... Attempting to upgrade from 4.7.12 to 4.7.14 John

Hi, Thanks for that... What is the actual sequence for converting a V4 application to V5? Take the following as an outline... Starting with my V4 development system with IN_DEV Disable IN_DEV (as I cannot upgrade with it set) Install V5 (IPS database is converted, etc...) Presumably my application is disabled by the upgrade as it is not compatible? Enable IN_DEV What do I have to do so that I can re-enable the application and start testing file by file? I don't believe that updating all files before I can test anything is practical. In many cases I'll have to play around and test various alternatives before I find something that might work under V5. What happens to all the hooks? Do they just get deleted by the upgrade to V5? (Obviously, I could retrieve them from my other V4 development system and use them as the basis for working out what to do under V5.) What happens to the MemberSync extension(s)? Do they just get deleted by the upgrade - so I just retrieve the code from my other V4 development system and insert it into a Listener? Are all me database tables preserved by the upgrade? Similarly, what happens to a customer/live system that runs my application and upgrades from V4 to V5? Starting with live system running my application on V4 IPS is upgraded from V4 to V5. Presumably my application is disabled by the upgrade as it is not compatible? 'Customer' installs the V5 compatible version of my application -- is all now OK with no loss of data? Thanks very much. John

Marc, Thanks for all replies. Will run the checker as advised. John

Marc, Thanks. As Invision 4.6.12.1 does not work with PHP 8.1 I am presuming that AdminCP would not work on that version so I cannot do an automatic upgrade. Correct? Does that mean that I have to do a "manual upgrade" following this outline: Maintenance mode Backup files + DB Switch to PHP 8.1 (current Invision installation no longer works at this point) Manual upgrade to latest Invision Normal mode Or, in full, to include FileRun in the sequence: Maintenance mode (both FileRun and Invision) Backups (both FileRun and Invision) Upgrade FileRun to latest (this includes the step to switch to PHP 8.1) Manual upgrade Invision to latest version Normal mode (both FileRun and Invision) Thanks. John